Jun 02 15:46:52.662432 ip-10-0-135-231 systemd[1]: Starting Kubernetes Kubelet...
Jun 02 15:46:53.168571 ip-10-0-135-231 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:53.168571 ip-10-0-135-231 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 02 15:46:53.168571 ip-10-0-135-231 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:53.168571 ip-10-0-135-231 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 02 15:46:53.168571 ip-10-0-135-231 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:53.170333 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.170243    2561 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 02 15:46:53.178308 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178283    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:53.178308 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178302    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:53.178308 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178307    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:53.178308 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178310    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:53.178308 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178314    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:53.178308 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178317    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178320    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178324    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178327    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178330    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178334    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178336    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178339    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178342    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178344    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178347    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178350    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178353    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178355    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178358    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178360    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178363    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178366    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178368    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178371    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:53.178527 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178373    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178376    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178385    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178388    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178391    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178393    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178396    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178398    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178401    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178403    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178406    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178408    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178411    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178413    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178417    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178419    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178422    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178427    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178431    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:53.179022 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178434    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178438    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178441    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178443    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178446    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178448    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178451    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178454    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178456    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178459    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178462    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178464    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178467    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178470    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178472    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178475    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178477    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178479    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178482    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178484    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:53.179594 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178487    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178489    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178492    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178494    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178496    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178499    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178501    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178505    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178507    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178512    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178515    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178519    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178521    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178524    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178533    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178537    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178540    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178542    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178545    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:53.180083 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178547    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178550    2561 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178552    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178974    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178980    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178983    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178986    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178989    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178993    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178996    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.178999    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179002    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179004    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179007    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179009    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179012    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179014    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179017    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179019    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179022    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:53.180562 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179025    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179028    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179031    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179033    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179036    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179039    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179042    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179050    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179053    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179056    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179058    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179061    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179063    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179066    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179068    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179071    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179073    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179075    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179079    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:53.181039 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179083    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179086    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179088    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179092    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179095    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179098    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179100    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179103    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179105    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179108    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179110    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179113    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179116    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179120    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179122    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179125    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179128    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179130    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179133    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179135    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:53.181533 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179138    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179146    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179149    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179152    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179154    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179170    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179173    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179176    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179178    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179181    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179183    2561 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179186    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179188    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179191    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179193    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179195    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179198    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179200    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179203    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179205    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:53.182028 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179208    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179211    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179214    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179216    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179219    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179222    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179224    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179227    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179229    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.179232    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180170    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180179    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180188    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180193    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180205    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180208    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180212    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180217    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180220    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180223    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180226    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 02 15:46:53.182536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180229    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180232    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180235    2561 flags.go:64] FLAG: --cgroup-root=""
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180239    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180242    2561 flags.go:64] FLAG: --client-ca-file=""
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180245    2561 flags.go:64] FLAG: --cloud-config=""
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180247    2561 flags.go:64] FLAG: --cloud-provider="external"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180250    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180261    2561 flags.go:64] FLAG: --cluster-domain=""
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180264    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180267    2561 flags.go:64] FLAG: --config-dir=""
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180270    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180273    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180278    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180281    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180283    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180287    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180290    2561 flags.go:64] FLAG: --contention-profiling="false"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180293    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180296    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180299    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180302    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180306    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180309    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180312    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 02 15:46:53.183057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180321    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180330    2561 flags.go:64] FLAG: --enable-server="true"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180333    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180340    2561 flags.go:64] FLAG: --event-burst="100"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180343    2561 flags.go:64] FLAG: --event-qps="50"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180346    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180350    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180352    2561 flags.go:64] FLAG: --eviction-hard=""
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180356    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180359    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180362    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180365    2561 flags.go:64] FLAG: --eviction-soft=""
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180368    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180372    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180374    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180377    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180380    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180382    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180385    2561 flags.go:64] FLAG: --feature-gates=""
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180389    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180392    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180395    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180399    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180402    2561 flags.go:64] FLAG: --healthz-port="10248"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180405    2561 flags.go:64] FLAG: --help="false"
Jun 02 15:46:53.183663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180408    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180411    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180414    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180416    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180420    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180423    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180425    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180428    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180431    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180439    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180443    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180446    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180448    2561 flags.go:64] FLAG: --kube-reserved=""
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180456    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180459    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180462    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180465    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180467    2561 flags.go:64] FLAG: --lock-file=""
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180470    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180473    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180476    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180481    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180484    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180487    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 02 15:46:53.184296 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180489    2561 flags.go:64] FLAG: --logging-format="text"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180492    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180495    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180498    2561 flags.go:64] FLAG: --manifest-url=""
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180501    2561 flags.go:64] FLAG: --manifest-url-header=""
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180510    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180513    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180517    2561 flags.go:64] FLAG: --max-pods="110"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180519    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180522    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180525    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180528    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180531    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180534    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180537    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180544    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180546    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180549    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180558    2561 flags.go:64] FLAG: --pod-cidr=""
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180561    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1083dc6abd9805f79797fa11c9accacc4a38caa073c94cb99097faffce05983"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180567    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180570    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180574    2561 flags.go:64] FLAG: --pods-per-core="0"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180577    2561 flags.go:64] FLAG: --port="10250"
Jun 02 15:46:53.184865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180580    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180583    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e41472ae722b9746"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180586    2561 flags.go:64] FLAG: --qos-reserved=""
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180589    2561 flags.go:64] FLAG: --read-only-port="10255"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180592    2561 flags.go:64] FLAG: --register-node="true"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180595    2561 flags.go:64] FLAG: --register-schedulable="true"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180598    2561 flags.go:64] FLAG: --register-with-taints=""
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180601    2561 flags.go:64] FLAG: --registry-burst="10"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180604    2561 flags.go:64] FLAG: --registry-qps="5"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180607    2561 flags.go:64] FLAG: --reserved-cpus=""
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180609    2561 flags.go:64] FLAG: --reserved-memory=""
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180613    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180616    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180620    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180622    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180625    2561 flags.go:64] FLAG: --runonce="false"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180628    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180631    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180634    2561 flags.go:64] FLAG: --seccomp-default="false"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180637    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180639    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180643    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180646    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180649    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180651    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180654    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 02 15:46:53.185474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180657    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180665    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180668    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180671    2561 flags.go:64] FLAG: --system-cgroups=""
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180674    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180679    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180682    2561 flags.go:64] FLAG: --tls-cert-file=""
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180685    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180693    2561 flags.go:64] FLAG: --tls-min-version=""
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180696    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180699    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180702    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180704    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180707    2561 flags.go:64] FLAG: --v="2"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180712    2561 flags.go:64] FLAG: --version="false"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180716    2561 flags.go:64] FLAG: --vmodule=""
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180720    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.180725    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180855    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180859    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180862    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180865    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180868    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:53.186114 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180871    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180874    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180877    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180879    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180882    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180884    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180887    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180889    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180892    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180895    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180897    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180906    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180909    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180911    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180914    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180916    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180919    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180922    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180924    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180927    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:53.186680 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180929    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180932    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180934    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180937    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180939    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180942    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180946    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180949    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180951    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180954    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180956    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180959    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180961    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180964    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180966    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180969    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180971    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180974    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180976    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180979    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:53.187235 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180981    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180984    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180986    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180989    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.180998    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181001    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181004    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181006    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181009    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181011    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181013    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181016    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181019    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181021    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181024    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181026    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181028    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181031    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181037    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:53.187721 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181040    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181044    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181048    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181051    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181054    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181057    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181060    2561 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181063    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181065    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181068    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181070    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181074    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181077    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181080    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181082    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181084    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181087    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181089    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181097    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:53.188217 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181100    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181103    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.181105    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.181729    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.188072    2561 server.go:530] "Kubelet version" kubeletVersion="v1.33.12"
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.188093    2561 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188142    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188148    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188151    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188155    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188173    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188176    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188179    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188182    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188186    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:53.188710 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188190    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188193    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188196    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188199    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188202    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188205    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188207    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188210    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188212    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188215    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188217    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188224    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188227    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188229    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188232    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188235    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188237    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188240    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188243    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:53.189089 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188246    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188248    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188251    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188255    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188258    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188261    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188264    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188267    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188269    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188272    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188275    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188277    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188280    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188282    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188285    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188287    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188290    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188293    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188295    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188298    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:53.189625 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188301    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188303    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188306    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188308    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188311    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188313    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188316    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188318    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188320    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188323    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188325    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188328    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188330    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188333    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188335    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188338    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188355    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188359    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188363    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:53.190108 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188366    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188369    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188371    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188374    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188377    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188379    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188382    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188384    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188387    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188389    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188393    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188395    2561 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188398    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188400    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188403    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188405    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188408    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188411    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:53.190578 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188413    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.188418    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188519    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188523    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188526    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188529    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188532    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188535    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188538    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188541    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188544    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188547    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188551    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188553    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188556    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188559    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:53.191026 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188561    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188564    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188566    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188568    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188571    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188573    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188576    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188579    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188581    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188584    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188586    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188589    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188591    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188594    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188596    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188599    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188601    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188604    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188606    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188608    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:53.191441 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188611    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188613    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188615    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188618    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188621    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188623    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188626    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188629    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188633    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188637    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188640    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188642    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188645    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188648    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188650    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188653    2561 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188656    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188658    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188661    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:53.191915 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188663    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188666    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188668    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188671    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188673    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188676    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188679    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188682    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188685    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188687    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188690    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188692    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188695    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188697    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188700    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188702    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188704    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188707    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188709    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:53.192401 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188712    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188715    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188718    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188720    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188723    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188726    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188728    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188731    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188733    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188736    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188739    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188741    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188743    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:53.188746    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.188750    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:53.192859 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.189691    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 02 15:46:53.193336 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.193290    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 02 15:46:53.194379 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.194367    2561 server.go:1019] "Starting client certificate rotation"
Jun 02 15:46:53.194481 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.194465    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:53.194516 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.194501    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:53.221129 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.221106    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:53.228976 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.228733    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:53.248489 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.248466    2561 log.go:25] "Validated CRI v1 runtime API"
Jun 02 15:46:53.254156 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.254138    2561 log.go:25] "Validated CRI v1 image API"
Jun 02 15:46:53.255403 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.255389    2561 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 02 15:46:53.261852 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.261825    2561 fs.go:135] Filesystem UUIDs: map[063d8ccf-7fb3-4a9e-8d54-9cd4640e2bda:/dev/nvme0n1p3 43f34084-ba95-4799-b823-b637ba87394a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jun 02 15:46:53.261927 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.261851    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 02 15:46:53.262879 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.262863    2561 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:53.266423 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.266304    2561 manager.go:217] Machine: {Timestamp:2026-06-02 15:46:53.2658094 +0000 UTC m=+0.460588892 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095731 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2199609d1d9c74e629c8d70a245a88 SystemUUID:ec219960-9d1d-9c74-e629-c8d70a245a88 BootID:2e556a94-963b-4add-a8cc-ebfb90700186 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2f:ed:5e:ac:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2f:ed:5e:ac:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:8c:b2:89:bc:8e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 02 15:46:53.266423 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.266417    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 02 15:46:53.266536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.266494    2561 manager.go:233] Version: {KernelVersion:5.14.0-570.117.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260521-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 02 15:46:53.269482 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.269461    2561 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 02 15:46:53.269635 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.269485    2561 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-231.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 02 15:46:53.269677 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.269645    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jun 02 15:46:53.269677 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.269652    2561 container_manager_linux.go:306] "Creating device plugin manager"
Jun 02 15:46:53.269677 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.269660    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:53.269677 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.269673    2561 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:53.270553 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.270543    2561 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:53.270699 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.270690    2561 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Jun 02 15:46:53.272926 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.272916    2561 kubelet.go:491] "Attempting to sync node with API server"
Jun 02 15:46:53.272973 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.272933    2561 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 02 15:46:53.272973 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.272954    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 02 15:46:53.272973 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.272968    2561 kubelet.go:397] "Adding apiserver pod source"
Jun 02 15:46:53.273091 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.272979    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 02 15:46:53.274449 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.274433    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:53.274519 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.274456    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:53.277984 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.277968    2561 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.12-3.rhaos4.20.git64c6a00.el9" apiVersion="v1"
Jun 02 15:46:53.279824 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.279812    2561 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 02 15:46:53.283154 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283140    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283180    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283189    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283195    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283204    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283210    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283223    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283229    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283236    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 02 15:46:53.283238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283242    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 02 15:46:53.283462 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283255    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 02 15:46:53.283462 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283261    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 02 15:46:53.283462 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283284    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 02 15:46:53.283462 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.283290    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 02 15:46:53.288824 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.288805    2561 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Jun 02 15:46:53.288916 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.288862    2561 server.go:1295] "Started kubelet"
Jun 02 15:46:53.288916 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.288895    2561 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-231.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 02 15:46:53.289000 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.288920    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 02 15:46:53.289000 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.288947    2561 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 02 15:46:53.289071 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.288999    2561 server_v1.go:47] "podresources" method="list" useActivePods=true
Jun 02 15:46:53.289135 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.289113    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 02 15:46:53.289284 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.289267    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 02 15:46:53.289654 ip-10-0-135-231 systemd[1]: Started Kubernetes Kubelet.
Jun 02 15:46:53.290808 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.290791    2561 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 02 15:46:53.291510 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.291498    2561 server.go:317] "Adding debug handlers to kubelet server"
Jun 02 15:46:53.296427 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.296408    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:53.297716 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.297697    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 02 15:46:53.297808 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.297789    2561 volume_manager.go:295] "The desired_state_of_world populator starts"
Jun 02 15:46:53.297808 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.297801    2561 volume_manager.go:297] "Starting Kubelet Volume Manager"
Jun 02 15:46:53.297941 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.297927    2561 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Jun 02 15:46:53.298067 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.298053    2561 reconstruct.go:97] "Volume reconstruction finished"
Jun 02 15:46:53.298136 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.298128    2561 reconciler.go:26] "Reconciler: start to sync state"
Jun 02 15:46:53.298533 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.297689    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-231.ec2.internal.18b54e3414c81106  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-231.ec2.internal,UID:ip-10-0-135-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-231.ec2.internal,},FirstTimestamp:2026-06-02 15:46:53.288820998 +0000 UTC m=+0.483600491,LastTimestamp:2026-06-02 15:46:53.288820998 +0000 UTC m=+0.483600491,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-231.ec2.internal,}"
Jun 02 15:46:53.298642 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.298631    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:53.299390 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.299369    2561 factory.go:55] Registering systemd factory
Jun 02 15:46:53.299390 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.299386    2561 factory.go:223] Registration of the systemd container factory successfully
Jun 02 15:46:53.299970 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.299852    2561 factory.go:153] Registering CRI-O factory
Jun 02 15:46:53.299970 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.299873    2561 factory.go:223] Registration of the crio container factory successfully
Jun 02 15:46:53.299970 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.299936    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 02 15:46:53.299970 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.299954    2561 factory.go:103] Registering Raw factory
Jun 02 15:46:53.299970 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.299968    2561 manager.go:1196] Started watching for new ooms in manager
Jun 02 15:46:53.300378 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.300361    2561 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 02 15:46:53.300638 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.300624    2561 manager.go:319] Starting recovery of all containers
Jun 02 15:46:53.303436 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.303408    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jun 02 15:46:53.303547 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.303524    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-231.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jun 02 15:46:53.311123 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.311010    2561 manager.go:324] Recovery completed
Jun 02 15:46:53.316766 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.316750    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:53.318599 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.318584    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:53.318663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.318610    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:53.318663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.318620    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:53.319045 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.319032    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 02 15:46:53.319095 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.319046    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 02 15:46:53.319095 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.319061    2561 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:53.321101 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.321039    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-231.ec2.internal.18b54e34168e7078  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-231.ec2.internal,UID:ip-10-0-135-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-231.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-231.ec2.internal,},FirstTimestamp:2026-06-02 15:46:53.318598776 +0000 UTC m=+0.513378266,LastTimestamp:2026-06-02 15:46:53.318598776 +0000 UTC m=+0.513378266,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-231.ec2.internal,}"
Jun 02 15:46:53.321557 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.321543    2561 policy_none.go:49] "None policy: Start"
Jun 02 15:46:53.321595 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.321566    2561 memory_manager.go:186] "Starting memorymanager" policy="None"
Jun 02 15:46:53.321595 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.321580    2561 state_mem.go:35] "Initializing new in-memory state store"
Jun 02 15:46:53.327689 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.327672    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zk6f7"
Jun 02 15:46:53.334843 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.334781    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-231.ec2.internal.18b54e34168eb0f7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-231.ec2.internal,UID:ip-10-0-135-231.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-231.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-231.ec2.internal,},FirstTimestamp:2026-06-02 15:46:53.318615287 +0000 UTC m=+0.513394779,LastTimestamp:2026-06-02 15:46:53.318615287 +0000 UTC m=+0.513394779,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-231.ec2.internal,}"
Jun 02 15:46:53.338808 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.338791    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zk6f7"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.359692    2561 manager.go:341] "Starting Device Plugin manager"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.359719    2561 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.359728    2561 server.go:85] "Starting device plugin registration server"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.359950    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.359962    2561 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.360049    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.360137    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.360146    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.360594    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 02 15:46:53.388558 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.360632    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:53.426668 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.426597    2561 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Jun 02 15:46:53.427809 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.427794    2561 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Jun 02 15:46:53.427889 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.427819    2561 status_manager.go:230] "Starting to sync pod status with apiserver"
Jun 02 15:46:53.427889 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.427839    2561 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 02 15:46:53.427889 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.427846    2561 kubelet.go:2451] "Starting kubelet main sync loop"
Jun 02 15:46:53.427889 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.427881    2561 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 02 15:46:53.429909 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.429889    2561 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:53.460575 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.460558    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:53.463340 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.463325    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:53.463400 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.463358    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:53.463400 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.463371    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:53.463400 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.463392    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.473288 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.473275    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.473331 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.473294    2561 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-231.ec2.internal\": node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:53.493267 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.493251    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:53.528243 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.528220    2561 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"]
Jun 02 15:46:53.528331 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.528282    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:53.529062 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.529044    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:53.529153 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.529070    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:53.529153 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.529085    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:53.530372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.530361    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:53.530538 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.530525    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.530573 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.530553    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:53.531055 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.531040    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:53.531114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.531068    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:53.531114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.531079    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:53.531204 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.531040    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:53.531204 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.531139    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:53.531204 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.531153    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:53.532486 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.532474    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.532527 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.532497    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:53.533198 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.533178    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:53.533292 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.533203    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:53.533292 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.533212    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:53.557781 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.557761    2561 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-231.ec2.internal\" not found" node="ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.562047 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.562034    2561 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-231.ec2.internal\" not found" node="ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.594087 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.594071    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:53.599341 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.599326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a6caceccfe170f02c366067007b6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"bd1a6caceccfe170f02c366067007b6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.599401 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.599351    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e787f9b1152ee0a0c6c9b3174979975-config\") pod \"kube-apiserver-proxy-ip-10-0-135-231.ec2.internal\" (UID: \"7e787f9b1152ee0a0c6c9b3174979975\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.599401 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.599372    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd1a6caceccfe170f02c366067007b6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"bd1a6caceccfe170f02c366067007b6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.694393 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.694319    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:53.699600 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.699586    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd1a6caceccfe170f02c366067007b6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"bd1a6caceccfe170f02c366067007b6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.699657 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.699608    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a6caceccfe170f02c366067007b6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"bd1a6caceccfe170f02c366067007b6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.699657 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.699626    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e787f9b1152ee0a0c6c9b3174979975-config\") pod \"kube-apiserver-proxy-ip-10-0-135-231.ec2.internal\" (UID: \"7e787f9b1152ee0a0c6c9b3174979975\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.699751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.699661    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7e787f9b1152ee0a0c6c9b3174979975-config\") pod \"kube-apiserver-proxy-ip-10-0-135-231.ec2.internal\" (UID: \"7e787f9b1152ee0a0c6c9b3174979975\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.699751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.699703    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd1a6caceccfe170f02c366067007b6e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"bd1a6caceccfe170f02c366067007b6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.699751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.699704    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd1a6caceccfe170f02c366067007b6e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal\" (UID: \"bd1a6caceccfe170f02c366067007b6e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.794950 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.794910    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:53.860327 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.860304    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.865833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:53.865812    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:53.895582 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.895557    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:53.996121 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:53.996057    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:54.096559 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:54.096528    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:54.194017 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.193993    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 02 15:46:54.194436 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.194128    2561 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:54.197153 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:54.197135    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:54.297216 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:54.297192    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:54.297216 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.297218    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:54.306776 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.306752    2561 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:54.313467 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.313449    2561 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:54.340549 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.340512    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-01 15:41:53 +0000 UTC" deadline="2028-03-05 07:00:39.700746578 +0000 UTC"
Jun 02 15:46:54.340549 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.340540    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15399h13m45.360208683s"
Jun 02 15:46:54.345529 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.345513    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6nl2"
Jun 02 15:46:54.356471 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.356448    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s6nl2"
Jun 02 15:46:54.366596 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.366581    2561 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:54.367074 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:54.367052    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1a6caceccfe170f02c366067007b6e.slice/crio-3e8c69a87f02b8d5c581ca8ad6901a70adbb2491e0e945b1d0ea85a02893e090 WatchSource:0}: Error finding container 3e8c69a87f02b8d5c581ca8ad6901a70adbb2491e0e945b1d0ea85a02893e090: Status 404 returned error can't find the container with id 3e8c69a87f02b8d5c581ca8ad6901a70adbb2491e0e945b1d0ea85a02893e090
Jun 02 15:46:54.371768 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.371754    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:46:54.376148 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:54.376125    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e787f9b1152ee0a0c6c9b3174979975.slice/crio-8dafbc23bb9f701d2dd66b2915b435f4b192c65532840c310f6dd648872b7e0e WatchSource:0}: Error finding container 8dafbc23bb9f701d2dd66b2915b435f4b192c65532840c310f6dd648872b7e0e: Status 404 returned error can't find the container with id 8dafbc23bb9f701d2dd66b2915b435f4b192c65532840c310f6dd648872b7e0e
Jun 02 15:46:54.399264 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:54.397894    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:54.431377 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.431331    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal" event={"ID":"7e787f9b1152ee0a0c6c9b3174979975","Type":"ContainerStarted","Data":"8dafbc23bb9f701d2dd66b2915b435f4b192c65532840c310f6dd648872b7e0e"}
Jun 02 15:46:54.432184 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.432149    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal" event={"ID":"bd1a6caceccfe170f02c366067007b6e","Type":"ContainerStarted","Data":"3e8c69a87f02b8d5c581ca8ad6901a70adbb2491e0e945b1d0ea85a02893e090"}
Jun 02 15:46:54.498336 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:54.498315    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:54.598680 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:54.598655    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-135-231.ec2.internal\" not found"
Jun 02 15:46:54.617727 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.617705    2561 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:54.698339 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.698303    2561 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:54.711995 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.711970    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:54.713065 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.713049    2561 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal"
Jun 02 15:46:54.725064 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:54.725040    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:55.274224 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.274200    2561 apiserver.go:52] "Watching apiserver"
Jun 02 15:46:55.283939 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.283912    2561 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 02 15:46:55.284398 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.284371    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hjmpt","kube-system/global-pull-secret-syncer-pn7nc","kube-system/konnectivity-agent-drrq4","kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc","openshift-dns/node-resolver-66lpd","openshift-image-registry/node-ca-zqnmk","openshift-network-diagnostics/network-check-target-rmnx9","openshift-network-operator/iptables-alerter-9fklx","openshift-cluster-node-tuning-operator/tuned-m7gh9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal","openshift-multus/multus-additional-cni-plugins-8htr9","openshift-multus/multus-qqwfk","openshift-multus/network-metrics-daemon-48g4z"]
Jun 02 15:46:55.286303 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.286282    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.287567 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.287550    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.289754 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.289731    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:46:55.290892 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.290864    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 02 15:46:55.290892 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.290872    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vsw8b\""
Jun 02 15:46:55.291393 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.291234    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 02 15:46:55.291393 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.291251    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 02 15:46:55.291393 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.291308    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 02 15:46:55.291393 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.291241    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:55.291627 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.291395    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:46:55.291683 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.291648    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s5vwk\""
Jun 02 15:46:55.291754 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.291736    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:55.292059 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.292041    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:55.292601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.292574    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.293243 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.293227    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 02 15:46:55.293415 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.293398    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 02 15:46:55.293602 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.293588    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2cnjs\""
Jun 02 15:46:55.296418 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.296400    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.297197 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.297073    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 02 15:46:55.297197 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.297186    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 02 15:46:55.297354 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.297206    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 02 15:46:55.297413 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.297385    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gnmp6\""
Jun 02 15:46:55.297819 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.297804    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.298825 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.298807    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 02 15:46:55.299149 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.299130    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.300519 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.300500    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.300602 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.300581    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:46:55.302755 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.302739    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.305041 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.304826    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pn2ph\""
Jun 02 15:46:55.305041 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.304867    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 02 15:46:55.305398 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.305378    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 02 15:46:55.305499 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.305479    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xcvq4\""
Jun 02 15:46:55.305647 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.305630    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:55.305713 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.305699    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 02 15:46:55.305767 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.305732    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lqclj\""
Jun 02 15:46:55.305817 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.305805    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 02 15:46:55.305865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.305380    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 02 15:46:55.306057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.306011    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:55.306988 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.306968    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 02 15:46:55.307080 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.306974    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 02 15:46:55.307273 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.307255    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 02 15:46:55.307431 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.307334    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 02 15:46:55.307498 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.307349    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 02 15:46:55.307563 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.307353    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9w8fp\""
Jun 02 15:46:55.307563 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.307537    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 02 15:46:55.308122 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308092    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysconfig\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.308226 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysctl-d\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.308226 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308153    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-system-cni-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308226 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-socket-dir-parent\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308366 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-cni-bin\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308366 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-cni-multus\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308366 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-conf-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308366 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-socket-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.308556 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308375    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5948a580-09d3-4717-aa41-4ed1de678f0d-serviceca\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.308556 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308405    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jvp\" (UniqueName: \"kubernetes.io/projected/5948a580-09d3-4717-aa41-4ed1de678f0d-kube-api-access-q2jvp\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.308556 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308429    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysctl-conf\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.308556 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308451    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-hostroot\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308556 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308472    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-multus-certs\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308556 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308492    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-etc-kubernetes\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308556 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308528    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-registration-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308563    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-kubernetes\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308592    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-os-release\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67e734df-160b-4056-8cfa-811074a6df62-cni-binary-copy\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-netns\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308696    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308714    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-582cs\" (UniqueName: \"kubernetes.io/projected/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-kube-api-access-582cs\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7581f7-059f-4573-aec7-312b2de9f73c-tmp-dir\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308761    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5948a580-09d3-4717-aa41-4ed1de678f0d-host\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.308767    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-run\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-host\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.308861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308833    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666zw\" (UniqueName: \"kubernetes.io/projected/dbd81b32-a9cc-4326-bcef-9275d9294f2e-kube-api-access-666zw\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308868    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-cni-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308906    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-k8s-cni-cncf-io\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-modprobe-d\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.308998    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-var-lib-kubelet\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309026    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67e734df-160b-4056-8cfa-811074a6df62-multus-daemon-config\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfv4\" (UniqueName: \"kubernetes.io/projected/67e734df-160b-4056-8cfa-811074a6df62-kube-api-access-sxfv4\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309072    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-device-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309094    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4944b44f-d4ce-45a7-8d53-08d3d71de354-host-slash\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-sys\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbd81b32-a9cc-4326-bcef-9275d9294f2e-tmp\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309224    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e-konnectivity-ca\") pod \"konnectivity-agent-drrq4\" (UID: \"cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e\") " pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309246    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-sys-fs\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4944b44f-d4ce-45a7-8d53-08d3d71de354-iptables-alerter-script\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f7581f7-059f-4573-aec7-312b2de9f73c-hosts-file\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.309387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309323    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77qkg\" (UniqueName: \"kubernetes.io/projected/1f7581f7-059f-4573-aec7-312b2de9f73c-kube-api-access-77qkg\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309401    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-lib-modules\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309425    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2rmb\" (UniqueName: \"kubernetes.io/projected/6d86be44-4451-4903-bf56-5179cec7698d-kube-api-access-q2rmb\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6qj9\" (UniqueName: \"kubernetes.io/projected/4944b44f-d4ce-45a7-8d53-08d3d71de354-kube-api-access-r6qj9\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-systemd\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309586    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-tuned\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309607    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-cnibin\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309630    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-kubelet\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.309953 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.309672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e-agent-certs\") pod \"konnectivity-agent-drrq4\" (UID: \"cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e\") " pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:46:55.310307 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.310205    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.314112 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.314093    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 02 15:46:55.314242 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.314112    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 02 15:46:55.314447 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.314432    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-p77xz\""
Jun 02 15:46:55.357739 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.357710    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:54 +0000 UTC" deadline="2027-11-18 06:53:13.658187643 +0000 UTC"
Jun 02 15:46:55.357739 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.357738    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12807h6m18.300451733s"
Jun 02 15:46:55.399893 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.399865    2561 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Jun 02 15:46:55.409985 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.409958    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-lib-modules\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.410142 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.409995    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:55.410142 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410026    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.410142 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410051    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-cnibin\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.410142 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410075    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e-agent-certs\") pod \"konnectivity-agent-drrq4\" (UID: \"cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e\") " pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:46:55.410142 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-cnibin\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.410142 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410137    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410144    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-run-netns\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410157    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-lib-modules\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.410184    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410156    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-cnibin\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410186    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovnkube-config\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.410277    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:46:55.910246803 +0000 UTC m=+3.105026298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410321    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysconfig\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410363    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysctl-d\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410408    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysconfig\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410406    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-system-cni-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410449    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-cni-bin\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.410465 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410467    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-system-cni-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410479    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-cni-bin\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410483    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-cni-multus\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410513    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-socket-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410514    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-cni-multus\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410537    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysctl-d\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-node-log\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-log-socket\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410606    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5948a580-09d3-4717-aa41-4ed1de678f0d-serviceca\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410628    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-socket-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410647    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-hostroot\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410656    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410695    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-multus-certs\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410673    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-multus-certs\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410731    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-hostroot\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410746    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-582cs\" (UniqueName: \"kubernetes.io/projected/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-kube-api-access-582cs\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410797    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-run-ovn-kubernetes\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.411047 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-cni-bin\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410891    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-kubernetes\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410916    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-netns\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgsl\" (UniqueName: \"kubernetes.io/projected/20a616f5-634d-4520-9fc3-5fe20beac101-kube-api-access-kdgsl\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410969    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410981    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-kubernetes\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.410999    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovn-node-metrics-cert\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411018    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5948a580-09d3-4717-aa41-4ed1de678f0d-serviceca\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411025    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5948a580-09d3-4717-aa41-4ed1de678f0d-host\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411042    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-netns\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411125    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-host\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411140    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5948a580-09d3-4717-aa41-4ed1de678f0d-host\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411193    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-666zw\" (UniqueName: \"kubernetes.io/projected/dbd81b32-a9cc-4326-bcef-9275d9294f2e-kube-api-access-666zw\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411205    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-host\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411219    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-k8s-cni-cncf-io\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411249    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411269    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-run-k8s-cni-cncf-io\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.411751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f77022b6-028e-4650-b07e-48525a6590f2-kubelet-config\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411303    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-cni-netd\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411326    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfv4\" (UniqueName: \"kubernetes.io/projected/67e734df-160b-4056-8cfa-811074a6df62-kube-api-access-sxfv4\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411347    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-device-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411371    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4944b44f-d4ce-45a7-8d53-08d3d71de354-host-slash\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411389    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-sys\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411398    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-device-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411403    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e-konnectivity-ca\") pod \"konnectivity-agent-drrq4\" (UID: \"cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e\") " pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411419    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-sys-fs\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411434    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4944b44f-d4ce-45a7-8d53-08d3d71de354-iptables-alerter-script\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411443    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-sys\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411460    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-os-release\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-env-overrides\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411497    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q2rmb\" (UniqueName: \"kubernetes.io/projected/6d86be44-4451-4903-bf56-5179cec7698d-kube-api-access-q2rmb\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411514    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qj9\" (UniqueName: \"kubernetes.io/projected/4944b44f-d4ce-45a7-8d53-08d3d71de354-kube-api-access-r6qj9\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411530    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411546    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-systemd\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.412485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411562    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-tuned\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411594    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-kubelet\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411612    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-cni-binary-copy\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411601    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-sys-fs\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f77022b6-028e-4650-b07e-48525a6590f2-dbus\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-systemd\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411662    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-registration-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411677    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-socket-dir-parent\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411692    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-conf-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-systemd-units\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovnkube-script-lib\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411740    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jvp\" (UniqueName: \"kubernetes.io/projected/5948a580-09d3-4717-aa41-4ed1de678f0d-kube-api-access-q2jvp\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411756    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysctl-conf\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411785    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-etc-kubernetes\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411802    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411433    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4944b44f-d4ce-45a7-8d53-08d3d71de354-host-slash\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-slash\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.413120 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-etc-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411867    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-systemd\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411880    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-ovn\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411910    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-os-release\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411936    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67e734df-160b-4056-8cfa-811074a6df62-cni-binary-copy\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411982    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e-konnectivity-ca\") pod \"konnectivity-agent-drrq4\" (UID: \"cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e\") " pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.411981    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-conf-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412043    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7581f7-059f-4573-aec7-312b2de9f73c-tmp-dir\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-system-cni-dir\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412099    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-os-release\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412107    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-var-lib-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412213    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-host-var-lib-kubelet\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412272    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-registration-dir\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412315    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-socket-dir-parent\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412345    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-run\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412385    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-cni-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.413836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412415    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412446    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-modprobe-d\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412466    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f7581f7-059f-4573-aec7-312b2de9f73c-tmp-dir\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412501    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-var-lib-kubelet\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412526    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67e734df-160b-4056-8cfa-811074a6df62-multus-daemon-config\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412554    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412593    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-etc-kubernetes\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412610    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-sysctl-conf\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412635    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-kubelet\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412666    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbd81b32-a9cc-4326-bcef-9275d9294f2e-tmp\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412693    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f7581f7-059f-4573-aec7-312b2de9f73c-hosts-file\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412725    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-77qkg\" (UniqueName: \"kubernetes.io/projected/1f7581f7-059f-4573-aec7-312b2de9f73c-kube-api-access-77qkg\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412754    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrpc\" (UniqueName: \"kubernetes.io/projected/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-kube-api-access-vwrpc\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412790    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67e734df-160b-4056-8cfa-811074a6df62-multus-cni-dir\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412814    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67e734df-160b-4056-8cfa-811074a6df62-cni-binary-copy\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412844    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-var-lib-kubelet\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412870    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-modprobe-d\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412695    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-etc-selinux\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.414601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412901    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/dbd81b32-a9cc-4326-bcef-9275d9294f2e-run\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.415317 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.412924    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1f7581f7-059f-4573-aec7-312b2de9f73c-hosts-file\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.415317 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.413297    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67e734df-160b-4056-8cfa-811074a6df62-multus-daemon-config\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.415317 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.413541    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4944b44f-d4ce-45a7-8d53-08d3d71de354-iptables-alerter-script\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.415317 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.414835    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e-agent-certs\") pod \"konnectivity-agent-drrq4\" (UID: \"cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e\") " pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:46:55.415317 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.415025    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/dbd81b32-a9cc-4326-bcef-9275d9294f2e-etc-tuned\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.415533 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.415319    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbd81b32-a9cc-4326-bcef-9275d9294f2e-tmp\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.429408 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.429384    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-582cs\" (UniqueName: \"kubernetes.io/projected/3001c2c6-e9c8-4abf-aa66-4cb8608c510b-kube-api-access-582cs\") pod \"aws-ebs-csi-driver-node-dl9pc\" (UID: \"3001c2c6-e9c8-4abf-aa66-4cb8608c510b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.429796 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.429777    2561 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:55.433324 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.433280    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77qkg\" (UniqueName: \"kubernetes.io/projected/1f7581f7-059f-4573-aec7-312b2de9f73c-kube-api-access-77qkg\") pod \"node-resolver-66lpd\" (UID: \"1f7581f7-059f-4573-aec7-312b2de9f73c\") " pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.433906 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.433884    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-666zw\" (UniqueName: \"kubernetes.io/projected/dbd81b32-a9cc-4326-bcef-9275d9294f2e-kube-api-access-666zw\") pod \"tuned-m7gh9\" (UID: \"dbd81b32-a9cc-4326-bcef-9275d9294f2e\") " pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.434213 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.434194    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfv4\" (UniqueName: \"kubernetes.io/projected/67e734df-160b-4056-8cfa-811074a6df62-kube-api-access-sxfv4\") pod \"multus-qqwfk\" (UID: \"67e734df-160b-4056-8cfa-811074a6df62\") " pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.434498 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.434472    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2rmb\" (UniqueName: \"kubernetes.io/projected/6d86be44-4451-4903-bf56-5179cec7698d-kube-api-access-q2rmb\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:55.434737 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.434721    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jvp\" (UniqueName: \"kubernetes.io/projected/5948a580-09d3-4717-aa41-4ed1de678f0d-kube-api-access-q2jvp\") pod \"node-ca-zqnmk\" (UID: \"5948a580-09d3-4717-aa41-4ed1de678f0d\") " pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.436454 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.436436    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qj9\" (UniqueName: \"kubernetes.io/projected/4944b44f-d4ce-45a7-8d53-08d3d71de354-kube-api-access-r6qj9\") pod \"iptables-alerter-9fklx\" (UID: \"4944b44f-d4ce-45a7-8d53-08d3d71de354\") " pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.513300 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513269    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513449 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513308    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-cni-binary-copy\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.513449 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513333    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f77022b6-028e-4650-b07e-48525a6590f2-dbus\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.513449 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513382    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-systemd\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513449 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513423    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-systemd-units\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513449 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513427    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-systemd\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513449 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513439    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovnkube-script-lib\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513459    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513480    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-slash\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513505    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-etc-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513529    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-ovn\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513542    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-systemd-units\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513557    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-system-cni-dir\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513591    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513600    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-etc-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513621    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-var-lib-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513634    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-run-ovn\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513653    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.513688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513685    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513711    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-kubelet\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513739    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrpc\" (UniqueName: \"kubernetes.io/projected/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-kube-api-access-vwrpc\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513767    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-system-cni-dir\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513740    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-slash\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513781    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-cnibin\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513815    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-run-netns\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513820    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-cnibin\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513842    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovnkube-config\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513870    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-var-lib-openvswitch\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513873    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-node-log\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513903    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-log-socket\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513934    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-run-netns\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513935    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513966    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-run-ovn-kubernetes\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513969    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovnkube-script-lib\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513992    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-cni-bin\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.513714    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:55.514132 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513987    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-cni-binary-copy\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514069    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-run-ovn-kubernetes\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.514075    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret podName:f77022b6-028e-4650-b07e-48525a6590f2 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:56.014042437 +0000 UTC m=+3.208821914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret") pod "global-pull-secret-syncer-pn7nc" (UID: "f77022b6-028e-4650-b07e-48525a6590f2") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514301    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-cni-bin\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513427    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f77022b6-028e-4650-b07e-48525a6590f2-dbus\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514401    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514401    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-kubelet\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.513906    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-node-log\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514437    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-log-socket\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514452    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/20a616f5-634d-4520-9fc3-5fe20beac101-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514497    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgsl\" (UniqueName: \"kubernetes.io/projected/20a616f5-634d-4520-9fc3-5fe20beac101-kube-api-access-kdgsl\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514528    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514553    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovn-node-metrics-cert\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514557    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovnkube-config\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514578    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514600    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f77022b6-028e-4650-b07e-48525a6590f2-kubelet-config\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.514938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514614    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.515608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514654    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f77022b6-028e-4650-b07e-48525a6590f2-kubelet-config\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:55.515608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514665    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-cni-netd\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.515608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514701    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-os-release\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.515608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514720    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.515608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514725    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-env-overrides\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.515608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514768    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-host-cni-netd\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.515608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.514807    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/20a616f5-634d-4520-9fc3-5fe20beac101-os-release\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.515857 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.515703    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-env-overrides\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.516778 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.516754    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-ovn-node-metrics-cert\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.528369 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.528315    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:55.528369 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.528340    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:55.528369 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.528353    2561 projected.go:194] Error preparing data for projected volume kube-api-access-xjf4j for pod openshift-network-diagnostics/network-check-target-rmnx9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:55.528584 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.528408    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j podName:5467a93a-b318-4861-b91b-11becab47db8 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:56.028395449 +0000 UTC m=+3.223174939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xjf4j" (UniqueName: "kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j") pod "network-check-target-rmnx9" (UID: "5467a93a-b318-4861-b91b-11becab47db8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:55.530522 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.530495    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrpc\" (UniqueName: \"kubernetes.io/projected/1f5684ac-ab95-4d17-9b1f-73d3bb1c8560-kube-api-access-vwrpc\") pod \"ovnkube-node-hjmpt\" (UID: \"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560\") " pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.530623 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.530553    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgsl\" (UniqueName: \"kubernetes.io/projected/20a616f5-634d-4520-9fc3-5fe20beac101-kube-api-access-kdgsl\") pod \"multus-additional-cni-plugins-8htr9\" (UID: \"20a616f5-634d-4520-9fc3-5fe20beac101\") " pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.598531 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.598498    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m7gh9"
Jun 02 15:46:55.606468 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.606445    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qqwfk"
Jun 02 15:46:55.615018 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.615002    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:46:55.620592 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.620571    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc"
Jun 02 15:46:55.627119 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.627102    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-66lpd"
Jun 02 15:46:55.634188 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.634151    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9fklx"
Jun 02 15:46:55.640690 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.640668    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zqnmk"
Jun 02 15:46:55.646324 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.646304    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:46:55.651931 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.651911    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8htr9"
Jun 02 15:46:55.888397 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.888363    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5948a580_09d3_4717_aa41_4ed1de678f0d.slice/crio-b529ce0c77afa9e3dc8fead24fbf51438a9091643696cc3aa6afa3625e2ca132 WatchSource:0}: Error finding container b529ce0c77afa9e3dc8fead24fbf51438a9091643696cc3aa6afa3625e2ca132: Status 404 returned error can't find the container with id b529ce0c77afa9e3dc8fead24fbf51438a9091643696cc3aa6afa3625e2ca132
Jun 02 15:46:55.889923 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.889746    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5684ac_ab95_4d17_9b1f_73d3bb1c8560.slice/crio-269fd11b04f0dabcaa23d4d7b41e2ad00a1ec0dba9a435f43370f05013a2f8bf WatchSource:0}: Error finding container 269fd11b04f0dabcaa23d4d7b41e2ad00a1ec0dba9a435f43370f05013a2f8bf: Status 404 returned error can't find the container with id 269fd11b04f0dabcaa23d4d7b41e2ad00a1ec0dba9a435f43370f05013a2f8bf
Jun 02 15:46:55.890906 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.890883    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc3ce25_2f60_4d5d_8e87_ef9f61e17e8e.slice/crio-93941c5c851efe4f61cf25b5eee69d1272e2f431bfdb4829cd0a7b7f925a8c41 WatchSource:0}: Error finding container 93941c5c851efe4f61cf25b5eee69d1272e2f431bfdb4829cd0a7b7f925a8c41: Status 404 returned error can't find the container with id 93941c5c851efe4f61cf25b5eee69d1272e2f431bfdb4829cd0a7b7f925a8c41
Jun 02 15:46:55.894311 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.894287    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f7581f7_059f_4573_aec7_312b2de9f73c.slice/crio-3c857dbeb4f24de146fdb500d5c0fc347fb3aa91c19cfefe5130a9737ae88391 WatchSource:0}: Error finding container 3c857dbeb4f24de146fdb500d5c0fc347fb3aa91c19cfefe5130a9737ae88391: Status 404 returned error can't find the container with id 3c857dbeb4f24de146fdb500d5c0fc347fb3aa91c19cfefe5130a9737ae88391
Jun 02 15:46:55.895348 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.895326    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a616f5_634d_4520_9fc3_5fe20beac101.slice/crio-54487452d7f0d2ef1bfddc2de37dd09b56c89466afd08ce7155e0b2afd61bab1 WatchSource:0}: Error finding container 54487452d7f0d2ef1bfddc2de37dd09b56c89466afd08ce7155e0b2afd61bab1: Status 404 returned error can't find the container with id 54487452d7f0d2ef1bfddc2de37dd09b56c89466afd08ce7155e0b2afd61bab1
Jun 02 15:46:55.896050 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.896029    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4944b44f_d4ce_45a7_8d53_08d3d71de354.slice/crio-6b8b179d0a2945b9560ae74d99ad30173a998b4ea01ef3f4ad834bb152b02e62 WatchSource:0}: Error finding container 6b8b179d0a2945b9560ae74d99ad30173a998b4ea01ef3f4ad834bb152b02e62: Status 404 returned error can't find the container with id 6b8b179d0a2945b9560ae74d99ad30173a998b4ea01ef3f4ad834bb152b02e62
Jun 02 15:46:55.896862 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.896815    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e734df_160b_4056_8cfa_811074a6df62.slice/crio-8c1524f70271abfca0587f3534eaa41a2fc1e22af8b05ecc5f10dbac61732512 WatchSource:0}: Error finding container 8c1524f70271abfca0587f3534eaa41a2fc1e22af8b05ecc5f10dbac61732512: Status 404 returned error can't find the container with id 8c1524f70271abfca0587f3534eaa41a2fc1e22af8b05ecc5f10dbac61732512
Jun 02 15:46:55.897865 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.897752    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3001c2c6_e9c8_4abf_aa66_4cb8608c510b.slice/crio-d3155d4de6dd2d056524b373b905fc2ab45145078b617e71975824d1d0b9f039 WatchSource:0}: Error finding container d3155d4de6dd2d056524b373b905fc2ab45145078b617e71975824d1d0b9f039: Status 404 returned error can't find the container with id d3155d4de6dd2d056524b373b905fc2ab45145078b617e71975824d1d0b9f039
Jun 02 15:46:55.900170 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:46:55.900071    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd81b32_a9cc_4326_bcef_9275d9294f2e.slice/crio-235368cfbd5de03124351ea16d46536eababcc83ef6438467008799083f6e00a WatchSource:0}: Error finding container 235368cfbd5de03124351ea16d46536eababcc83ef6438467008799083f6e00a: Status 404 returned error can't find the container with id 235368cfbd5de03124351ea16d46536eababcc83ef6438467008799083f6e00a
Jun 02 15:46:55.917353 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:55.917327    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:55.917489 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.917469    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:55.917593 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:55.917523    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:46:56.917505924 +0000 UTC m=+4.112285402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:56.018074 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.017937    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:56.018194 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.018072    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:56.018194 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.018135    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret podName:f77022b6-028e-4650-b07e-48525a6590f2 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:57.018120967 +0000 UTC m=+4.212900450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret") pod "global-pull-secret-syncer-pn7nc" (UID: "f77022b6-028e-4650-b07e-48525a6590f2") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:56.119081 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.119027    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:46:56.119264 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.119148    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:56.119264 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.119188    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:56.119264 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.119202    2561 projected.go:194] Error preparing data for projected volume kube-api-access-xjf4j for pod openshift-network-diagnostics/network-check-target-rmnx9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:56.119264 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.119253    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j podName:5467a93a-b318-4861-b91b-11becab47db8 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:57.119240203 +0000 UTC m=+4.314019682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjf4j" (UniqueName: "kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j") pod "network-check-target-rmnx9" (UID: "5467a93a-b318-4861-b91b-11becab47db8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:56.358294 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.358141    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:54 +0000 UTC" deadline="2027-11-24 21:08:26.006546888 +0000 UTC"
Jun 02 15:46:56.358294 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.358200    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12965h21m29.648350813s"
Jun 02 15:46:56.431348 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.431272    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:46:56.431502 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.431394    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:46:56.445702 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.445674    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-drrq4" event={"ID":"cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e","Type":"ContainerStarted","Data":"93941c5c851efe4f61cf25b5eee69d1272e2f431bfdb4829cd0a7b7f925a8c41"}
Jun 02 15:46:56.448467 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.448442    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7gh9" event={"ID":"dbd81b32-a9cc-4326-bcef-9275d9294f2e","Type":"ContainerStarted","Data":"235368cfbd5de03124351ea16d46536eababcc83ef6438467008799083f6e00a"}
Jun 02 15:46:56.451651 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.451616    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"269fd11b04f0dabcaa23d4d7b41e2ad00a1ec0dba9a435f43370f05013a2f8bf"}
Jun 02 15:46:56.462290 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.462247    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zqnmk" event={"ID":"5948a580-09d3-4717-aa41-4ed1de678f0d","Type":"ContainerStarted","Data":"b529ce0c77afa9e3dc8fead24fbf51438a9091643696cc3aa6afa3625e2ca132"}
Jun 02 15:46:56.467477 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.466733    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal" event={"ID":"7e787f9b1152ee0a0c6c9b3174979975","Type":"ContainerStarted","Data":"9c7b213311d2df450b255bce35521200452679cede15586c88aac88c5e666a31"}
Jun 02 15:46:56.481686 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.481188    2561 generic.go:358] "Generic (PLEG): container finished" podID="bd1a6caceccfe170f02c366067007b6e" containerID="dd759b8541451a358dd45f59abf544a60e6ae331527b984d359cd445a7c4f83f" exitCode=0
Jun 02 15:46:56.481686 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.481266    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal" event={"ID":"bd1a6caceccfe170f02c366067007b6e","Type":"ContainerDied","Data":"dd759b8541451a358dd45f59abf544a60e6ae331527b984d359cd445a7c4f83f"}
Jun 02 15:46:56.488391 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.488101    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-231.ec2.internal" podStartSLOduration=2.488087085 podStartE2EDuration="2.488087085s" podCreationTimestamp="2026-06-02 15:46:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:56.487321474 +0000 UTC m=+3.682100975" watchObservedRunningTime="2026-06-02 15:46:56.488087085 +0000 UTC m=+3.682866582"
Jun 02 15:46:56.513751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.513700    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc" event={"ID":"3001c2c6-e9c8-4abf-aa66-4cb8608c510b","Type":"ContainerStarted","Data":"d3155d4de6dd2d056524b373b905fc2ab45145078b617e71975824d1d0b9f039"}
Jun 02 15:46:56.528524 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.528484    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqwfk" event={"ID":"67e734df-160b-4056-8cfa-811074a6df62","Type":"ContainerStarted","Data":"8c1524f70271abfca0587f3534eaa41a2fc1e22af8b05ecc5f10dbac61732512"}
Jun 02 15:46:56.545265 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.545058    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9fklx" event={"ID":"4944b44f-d4ce-45a7-8d53-08d3d71de354","Type":"ContainerStarted","Data":"6b8b179d0a2945b9560ae74d99ad30173a998b4ea01ef3f4ad834bb152b02e62"}
Jun 02 15:46:56.556007 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.555978    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8htr9" event={"ID":"20a616f5-634d-4520-9fc3-5fe20beac101","Type":"ContainerStarted","Data":"54487452d7f0d2ef1bfddc2de37dd09b56c89466afd08ce7155e0b2afd61bab1"}
Jun 02 15:46:56.575146 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.575107    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66lpd" event={"ID":"1f7581f7-059f-4573-aec7-312b2de9f73c","Type":"ContainerStarted","Data":"3c857dbeb4f24de146fdb500d5c0fc347fb3aa91c19cfefe5130a9737ae88391"}
Jun 02 15:46:56.926342 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:56.925764    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:56.926342 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.925920    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:56.926342 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:56.925983    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:46:58.925964973 +0000 UTC m=+6.120744458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:57.027533 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:57.026918    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:57.027533 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:57.027061    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:57.027533 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:57.027122    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret podName:f77022b6-028e-4650-b07e-48525a6590f2 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:59.027103735 +0000 UTC m=+6.221883218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret") pod "global-pull-secret-syncer-pn7nc" (UID: "f77022b6-028e-4650-b07e-48525a6590f2") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:57.128009 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:57.127972    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:46:57.128212 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:57.128200    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:57.128281 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:57.128220    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:57.128281 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:57.128233    2561 projected.go:194] Error preparing data for projected volume kube-api-access-xjf4j for pod openshift-network-diagnostics/network-check-target-rmnx9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:57.128378 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:57.128290    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j podName:5467a93a-b318-4861-b91b-11becab47db8 nodeName:}" failed. No retries permitted until 2026-06-02 15:46:59.128272527 +0000 UTC m=+6.323052010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjf4j" (UniqueName: "kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j") pod "network-check-target-rmnx9" (UID: "5467a93a-b318-4861-b91b-11becab47db8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:57.431219 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:57.431113    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:57.431661 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:57.431271    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:46:57.431726 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:57.431675    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:57.431880 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:57.431842    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:46:57.585860 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:57.585826    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal" event={"ID":"bd1a6caceccfe170f02c366067007b6e","Type":"ContainerStarted","Data":"1da9716b108331c05a1f92e57204d0865faa88133970a7c6efca590c7b0f10d6"}
Jun 02 15:46:57.602145 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:57.601476    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-231.ec2.internal" podStartSLOduration=3.601459629 podStartE2EDuration="3.601459629s" podCreationTimestamp="2026-06-02 15:46:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:57.601202734 +0000 UTC m=+4.795982238" watchObservedRunningTime="2026-06-02 15:46:57.601459629 +0000 UTC m=+4.796239133"
Jun 02 15:46:58.428934 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:58.428903    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:46:58.429117 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:58.429024    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:46:58.945060 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:58.945016    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:58.945523 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:58.945209    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:58.945523 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:58.945277    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:02.945256337 +0000 UTC m=+10.140035819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:59.045820 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:59.045453    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:59.045820 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:59.045629    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:59.045820 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:59.045702    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret podName:f77022b6-028e-4650-b07e-48525a6590f2 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:03.045683198 +0000 UTC m=+10.240462682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret") pod "global-pull-secret-syncer-pn7nc" (UID: "f77022b6-028e-4650-b07e-48525a6590f2") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:46:59.146097 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:59.146056    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:46:59.146286 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:59.146265    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:59.146395 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:59.146290    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:59.146395 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:59.146305    2561 projected.go:194] Error preparing data for projected volume kube-api-access-xjf4j for pod openshift-network-diagnostics/network-check-target-rmnx9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:59.146395 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:59.146366    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j podName:5467a93a-b318-4861-b91b-11becab47db8 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:03.146347666 +0000 UTC m=+10.341127163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjf4j" (UniqueName: "kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j") pod "network-check-target-rmnx9" (UID: "5467a93a-b318-4861-b91b-11becab47db8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:59.428835 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:59.428758    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:46:59.428977 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:59.428896    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:46:59.429337 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:46:59.429321    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:46:59.429438 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:46:59.429420    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:00.429107 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:00.429070    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:00.429562 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:00.429218    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:01.429187 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:01.428595    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:01.429187 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:01.428636    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:01.429187 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:01.428763    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:01.429187 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:01.428930    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:02.428761 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:02.428726    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:02.428930 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:02.428866    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:02.978538 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:02.978504    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:02.979014 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:02.978659    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:02.979014 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:02.978720    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:10.978704636 +0000 UTC m=+18.173484113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:03.079819 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:03.079698    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:03.079979 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:03.079873    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:03.079979 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:03.079950    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret podName:f77022b6-028e-4650-b07e-48525a6590f2 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:11.079929593 +0000 UTC m=+18.274709073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret") pod "global-pull-secret-syncer-pn7nc" (UID: "f77022b6-028e-4650-b07e-48525a6590f2") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:03.180990 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:03.180958    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:03.181192 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:03.181154    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:03.181192 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:03.181195    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:03.181382 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:03.181205    2561 projected.go:194] Error preparing data for projected volume kube-api-access-xjf4j for pod openshift-network-diagnostics/network-check-target-rmnx9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:03.181382 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:03.181260    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j podName:5467a93a-b318-4861-b91b-11becab47db8 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:11.181243009 +0000 UTC m=+18.376022500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjf4j" (UniqueName: "kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j") pod "network-check-target-rmnx9" (UID: "5467a93a-b318-4861-b91b-11becab47db8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:03.430090 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:03.430014    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:03.430269 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:03.430147    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:03.430269 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:03.430207    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:03.430379 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:03.430316    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:04.428370 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:04.428332    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:04.428832 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:04.428459    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:05.428669 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:05.428630    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:05.429121 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:05.428780    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:05.429233 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:05.428630    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:05.429341 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:05.429322    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:06.428587 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:06.428553    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:06.428762 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:06.428657    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:07.428653 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:07.428608    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:07.428819 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:07.428788    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:07.429197 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:07.428866    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:07.429197 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:07.428999    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:08.428688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:08.428657    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:08.428861 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:08.428775    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:09.428657 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:09.428626    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:09.428902 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:09.428727    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:09.428902 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:09.428759    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:09.428902 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:09.428817    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:10.429045 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:10.429009    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:10.429559 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:10.429138    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:11.042123 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:11.042087    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:11.042313 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.042268    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:11.042372 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.042349    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:27.042326129 +0000 UTC m=+34.237105621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:11.143054 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:11.143017    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:11.143231 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.143175    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:11.143289 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.143238    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret podName:f77022b6-028e-4650-b07e-48525a6590f2 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:27.143221492 +0000 UTC m=+34.338000976 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret") pod "global-pull-secret-syncer-pn7nc" (UID: "f77022b6-028e-4650-b07e-48525a6590f2") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:11.243387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:11.243343    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:11.243567 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.243517    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:11.243567 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.243541    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:11.243567 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.243555    2561 projected.go:194] Error preparing data for projected volume kube-api-access-xjf4j for pod openshift-network-diagnostics/network-check-target-rmnx9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:11.243716 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.243610    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j podName:5467a93a-b318-4861-b91b-11becab47db8 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:27.243594002 +0000 UTC m=+34.438373500 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjf4j" (UniqueName: "kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j") pod "network-check-target-rmnx9" (UID: "5467a93a-b318-4861-b91b-11becab47db8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:11.428350 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:11.428312    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:11.428498 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:11.428321    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:11.428498 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.428433    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:11.428574 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:11.428509    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:12.428710 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:12.428672    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:12.429127 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:12.428804    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:13.429312 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:13.429281    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:13.429766 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:13.429387    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:13.429766 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:13.429441    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:13.429766 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:13.429508    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:14.429055 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.428798    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:14.429219 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:14.429188    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:14.619962 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.619933    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-drrq4" event={"ID":"cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e","Type":"ContainerStarted","Data":"4dc79e91e7092da2a8a673a23be240f8bae5d1faccc394922f7f459af889a782"}
Jun 02 15:47:14.621148 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.621126    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m7gh9" event={"ID":"dbd81b32-a9cc-4326-bcef-9275d9294f2e","Type":"ContainerStarted","Data":"2e0df128113e046d68d894c22d33773b07f159ee95797b38d73db7635e431726"}
Jun 02 15:47:14.622205 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.622185    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"58756104e88db215b6fd47e5bd28d503969af2c6d4fb6186fc94043687696458"}
Jun 02 15:47:14.623259 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.623239    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zqnmk" event={"ID":"5948a580-09d3-4717-aa41-4ed1de678f0d","Type":"ContainerStarted","Data":"ac6bba4961163ed32e4d349bb3ab7c701eb14dd219a3b2ca5d1275172a1731e6"}
Jun 02 15:47:14.624308 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.624289    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc" event={"ID":"3001c2c6-e9c8-4abf-aa66-4cb8608c510b","Type":"ContainerStarted","Data":"a824223559e3d5d2e3ac0f5783a11fdd9751b86037475da61caf8ad222334387"}
Jun 02 15:47:14.625450 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.625429    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qqwfk" event={"ID":"67e734df-160b-4056-8cfa-811074a6df62","Type":"ContainerStarted","Data":"ff4b6ce8afadcef524e929769af75311209762b5bb23174309c9477c1addebb8"}
Jun 02 15:47:14.626742 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.626722    2561 generic.go:358] "Generic (PLEG): container finished" podID="20a616f5-634d-4520-9fc3-5fe20beac101" containerID="ecda25f181a7449bd6deddd4a8388c7e3f75cee4e7870e3ec71aa958e2368e63" exitCode=0
Jun 02 15:47:14.626834 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.626796    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8htr9" event={"ID":"20a616f5-634d-4520-9fc3-5fe20beac101","Type":"ContainerDied","Data":"ecda25f181a7449bd6deddd4a8388c7e3f75cee4e7870e3ec71aa958e2368e63"}
Jun 02 15:47:14.628083 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.628044    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-66lpd" event={"ID":"1f7581f7-059f-4573-aec7-312b2de9f73c","Type":"ContainerStarted","Data":"f0c6d1944a0b87c0e6509275dcd0856873b3e61193ca1499661e6ace69c20426"}
Jun 02 15:47:14.647208 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.647150    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-drrq4" podStartSLOduration=3.480961675 podStartE2EDuration="21.6471412s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.892918977 +0000 UTC m=+3.087698462" lastFinishedPulling="2026-06-02 15:47:14.059098496 +0000 UTC m=+21.253877987" observedRunningTime="2026-06-02 15:47:14.646652452 +0000 UTC m=+21.841431951" watchObservedRunningTime="2026-06-02 15:47:14.6471412 +0000 UTC m=+21.841920699"
Jun 02 15:47:14.705803 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.705765    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m7gh9" podStartSLOduration=3.547835746 podStartE2EDuration="21.705751306s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.902652316 +0000 UTC m=+3.097431799" lastFinishedPulling="2026-06-02 15:47:14.060567866 +0000 UTC m=+21.255347359" observedRunningTime="2026-06-02 15:47:14.685771904 +0000 UTC m=+21.880551410" watchObservedRunningTime="2026-06-02 15:47:14.705751306 +0000 UTC m=+21.900530798"
Jun 02 15:47:14.723673 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.723623    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-66lpd" podStartSLOduration=3.560401118 podStartE2EDuration="21.723607879s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.895946988 +0000 UTC m=+3.090726472" lastFinishedPulling="2026-06-02 15:47:14.059153677 +0000 UTC m=+21.253933233" observedRunningTime="2026-06-02 15:47:14.722565251 +0000 UTC m=+21.917344756" watchObservedRunningTime="2026-06-02 15:47:14.723607879 +0000 UTC m=+21.918387381"
Jun 02 15:47:14.724127 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:14.724090    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-qqwfk" podStartSLOduration=3.551276652 podStartE2EDuration="21.724080972s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.899938797 +0000 UTC m=+3.094718289" lastFinishedPulling="2026-06-02 15:47:14.072743131 +0000 UTC m=+21.267522609" observedRunningTime="2026-06-02 15:47:14.706490458 +0000 UTC m=+21.901269958" watchObservedRunningTime="2026-06-02 15:47:14.724080972 +0000 UTC m=+21.918860476"
Jun 02 15:47:15.428527 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.428282    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:15.428684 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.428487    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:15.428684 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:15.428635    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:15.428799 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:15.428691    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:15.483978 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.483731    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 02 15:47:15.631510 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.631464    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9fklx" event={"ID":"4944b44f-d4ce-45a7-8d53-08d3d71de354","Type":"ContainerStarted","Data":"ecfaa8bca5bf0f003fadcd56176245094362691e4d26043e1542f245c89e14de"}
Jun 02 15:47:15.634264 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.634245    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 15:47:15.634635 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.634607    2561 generic.go:358] "Generic (PLEG): container finished" podID="1f5684ac-ab95-4d17-9b1f-73d3bb1c8560" containerID="3566e42dbb07ff4f86eb603d81ef9e07222ebe929fe1bbaebe0778f9846fdc7a" exitCode=1
Jun 02 15:47:15.634757 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.634661    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"76f97b9bb8acec41e22d22ff821e706432716695d113dbd5fe65b4d9bd14c180"}
Jun 02 15:47:15.634757 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.634696    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"39ec1ee9f2bb51a2b4f5eae514c33b1dc163070db180265763b9299aad7ae385"}
Jun 02 15:47:15.634757 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.634710    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"cced98376da1437f057e9701c2d273a45fff09784cac8141f3c4c847c857bc3e"}
Jun 02 15:47:15.634757 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.634724    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"aaca2ebae3403166f6935ffd038dc858a352f1d7ac7248ce4a9f04586b0ebb40"}
Jun 02 15:47:15.634757 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.634736    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerDied","Data":"3566e42dbb07ff4f86eb603d81ef9e07222ebe929fe1bbaebe0778f9846fdc7a"}
Jun 02 15:47:15.636257 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.636213    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc" event={"ID":"3001c2c6-e9c8-4abf-aa66-4cb8608c510b","Type":"ContainerStarted","Data":"d91d4396def0af33740386fe6234c9b3d7c188094794145cbb21cf7292949b8f"}
Jun 02 15:47:15.665612 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.665561    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9fklx" podStartSLOduration=4.503762927 podStartE2EDuration="22.665548901s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.898803572 +0000 UTC m=+3.093583066" lastFinishedPulling="2026-06-02 15:47:14.06058956 +0000 UTC m=+21.255369040" observedRunningTime="2026-06-02 15:47:15.66492663 +0000 UTC m=+22.859706131" watchObservedRunningTime="2026-06-02 15:47:15.665548901 +0000 UTC m=+22.860328401"
Jun 02 15:47:15.666261 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:15.666226    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zqnmk" podStartSLOduration=12.739584682 podStartE2EDuration="22.666217143s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.890074977 +0000 UTC m=+3.084854459" lastFinishedPulling="2026-06-02 15:47:05.816707439 +0000 UTC m=+13.011486920" observedRunningTime="2026-06-02 15:47:14.75989142 +0000 UTC m=+21.954670920" watchObservedRunningTime="2026-06-02 15:47:15.666217143 +0000 UTC m=+22.860996643"
Jun 02 15:47:16.372356 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:16.372254    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-02T15:47:15.483746856Z","UUID":"3ac2900f-c1fc-4898-b457-6b9097d9ce7f","Handler":null,"Name":"","Endpoint":""}
Jun 02 15:47:16.375701 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:16.375676    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 02 15:47:16.375701 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:16.375705    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 02 15:47:16.428685 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:16.428632    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:16.428792 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:16.428739    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:16.496341 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:16.496317    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:47:16.496841 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:16.496820    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:47:16.640524 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:16.640490    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc" event={"ID":"3001c2c6-e9c8-4abf-aa66-4cb8608c510b","Type":"ContainerStarted","Data":"d142da2022beebf42b10dc1f343496d1382e2767480b1652b7da0ac4afbd36a9"}
Jun 02 15:47:16.666098 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:16.666044    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dl9pc" podStartSLOduration=3.159073507 podStartE2EDuration="23.666031662s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.901833452 +0000 UTC m=+3.096612931" lastFinishedPulling="2026-06-02 15:47:16.408791593 +0000 UTC m=+23.603571086" observedRunningTime="2026-06-02 15:47:16.665918987 +0000 UTC m=+23.860698483" watchObservedRunningTime="2026-06-02 15:47:16.666031662 +0000 UTC m=+23.860811162"
Jun 02 15:47:17.428974 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:17.428948    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:17.429088 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:17.428944    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:17.429088 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:17.429067    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:17.429225 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:17.429208    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:17.646008 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:17.645811    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 15:47:17.646437 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:17.646402    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"6549ad745261363c4d81a96f126e976d6e5534c7c530bc871d277c756e741fa1"}
Jun 02 15:47:17.646437 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:17.646427    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jun 02 15:47:18.445601 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:18.445569    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:18.445768 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:18.445673    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:19.428040 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.428009    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:19.428589 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.428044    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:19.428589 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:19.428113    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:19.428589 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:19.428188    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:19.652817 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.652559    2561 generic.go:358] "Generic (PLEG): container finished" podID="20a616f5-634d-4520-9fc3-5fe20beac101" containerID="692f9051e52b4f2596705368f7f9d3c6cdfcd698ef051d3926d3e8f063cd1dbe" exitCode=0
Jun 02 15:47:19.652968 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.652654    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8htr9" event={"ID":"20a616f5-634d-4520-9fc3-5fe20beac101","Type":"ContainerDied","Data":"692f9051e52b4f2596705368f7f9d3c6cdfcd698ef051d3926d3e8f063cd1dbe"}
Jun 02 15:47:19.656049 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.655968    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 15:47:19.656279 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.656259    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"031658d0e6226025a53795021ae97f575bab7f2a14a1b54b1ca55bfce83750d0"}
Jun 02 15:47:19.656591 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.656564    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:47:19.656718 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.656600    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:47:19.656776 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.656722    2561 scope.go:117] "RemoveContainer" containerID="3566e42dbb07ff4f86eb603d81ef9e07222ebe929fe1bbaebe0778f9846fdc7a"
Jun 02 15:47:19.672739 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.672719    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:47:19.673018 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:19.673001    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:47:20.428527 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:20.428495    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:20.428972 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:20.428600    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:20.660474 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:20.660438    2561 generic.go:358] "Generic (PLEG): container finished" podID="20a616f5-634d-4520-9fc3-5fe20beac101" containerID="3809ba6330fb049dd0077f9b690cdf89020c94f3801751041d322d1469acee79" exitCode=0
Jun 02 15:47:20.660636 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:20.660520    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8htr9" event={"ID":"20a616f5-634d-4520-9fc3-5fe20beac101","Type":"ContainerDied","Data":"3809ba6330fb049dd0077f9b690cdf89020c94f3801751041d322d1469acee79"}
Jun 02 15:47:20.664349 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:20.664332    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 15:47:20.664687 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:20.664646    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" event={"ID":"1f5684ac-ab95-4d17-9b1f-73d3bb1c8560","Type":"ContainerStarted","Data":"8d77664742051a14dd2f5a1f07b13a7ce86324f3a10abd5e09d427866f55fc60"}
Jun 02 15:47:20.664820 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:20.664803    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jun 02 15:47:20.739305 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:20.739260    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt" podStartSLOduration=9.165500564 podStartE2EDuration="27.739244592s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.8915013 +0000 UTC m=+3.086280781" lastFinishedPulling="2026-06-02 15:47:14.465245331 +0000 UTC m=+21.660024809" observedRunningTime="2026-06-02 15:47:20.734929093 +0000 UTC m=+27.929708595" watchObservedRunningTime="2026-06-02 15:47:20.739244592 +0000 UTC m=+27.934024074"
Jun 02 15:47:21.428815 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.428788    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:21.429087 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.428817    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:21.429087 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:21.428919    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:21.429087 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:21.429012    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:21.443617 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.443592    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rmnx9"]
Jun 02 15:47:21.443695 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.443681    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:21.443767 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:21.443751    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:21.445402 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.445372    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pn7nc"]
Jun 02 15:47:21.455903 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.455877    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48g4z"]
Jun 02 15:47:21.668357 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.668324    2561 generic.go:358] "Generic (PLEG): container finished" podID="20a616f5-634d-4520-9fc3-5fe20beac101" containerID="8e9a618cf205069f50ef693dc38d0c219d2163e28fd4cf7dd8676e823ee60d9b" exitCode=0
Jun 02 15:47:21.668522 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.668403    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8htr9" event={"ID":"20a616f5-634d-4520-9fc3-5fe20beac101","Type":"ContainerDied","Data":"8e9a618cf205069f50ef693dc38d0c219d2163e28fd4cf7dd8676e823ee60d9b"}
Jun 02 15:47:21.668522 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.668498    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:21.668608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.668562    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jun 02 15:47:21.668697 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:21.668671    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:21.669095 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:21.668989    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:21.669095 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:21.669067    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:22.875530 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:22.875326    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:47:22.875977 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:22.875657    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jun 02 15:47:22.875977 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:22.875938    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-drrq4"
Jun 02 15:47:23.428961 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:23.428935    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:23.429139 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:23.429018    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:23.429139 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:23.429054    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:23.429139 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:23.429116    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:23.429321 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:23.429133    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:23.429321 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:23.429264    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:25.428377 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:25.428340    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:25.428938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:25.428339    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:25.428938 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:25.428476    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:47:25.428938 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:25.428547    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rmnx9" podUID="5467a93a-b318-4861-b91b-11becab47db8"
Jun 02 15:47:25.428938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:25.428342    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:25.428938 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:25.428626    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pn7nc" podUID="f77022b6-028e-4650-b07e-48525a6590f2"
Jun 02 15:47:25.461541 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:25.461515    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:47:25.461746 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:25.461733    2561 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jun 02 15:47:25.482587 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:25.482565    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hjmpt"
Jun 02 15:47:27.068328 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.068241    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:27.068837 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.068377    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:27.068837 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.068431    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:59.068418585 +0000 UTC m=+66.263198069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:27.109045 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.109019    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-135-231.ec2.internal" event="NodeReady"
Jun 02 15:47:27.109212 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.109177    2561 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Jun 02 15:47:27.158499 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.157050    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-659f487f9f-qbcmr"]
Jun 02 15:47:27.169642 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.169603    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:27.169801 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.169725    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:27.169801 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.169783    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret podName:f77022b6-028e-4650-b07e-48525a6590f2 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:59.16976652 +0000 UTC m=+66.364545998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret") pod "global-pull-secret-syncer-pn7nc" (UID: "f77022b6-028e-4650-b07e-48525a6590f2") : object "kube-system"/"original-pull-secret" not registered
Jun 02 15:47:27.184421 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.184396    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7tj9z"]
Jun 02 15:47:27.184573 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.184546    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.188915 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.188886    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jun 02 15:47:27.188915 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.188913    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kkg82\""
Jun 02 15:47:27.189145 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.189132    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jun 02 15:47:27.189295 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.189279    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jun 02 15:47:27.197484 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.197448    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jun 02 15:47:27.200609 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.200541    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659f487f9f-qbcmr"]
Jun 02 15:47:27.200722 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.200634    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w2q49"]
Jun 02 15:47:27.200722 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.200666    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.205117 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.205097    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 02 15:47:27.205695 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.205677    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 02 15:47:27.205779 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.205700    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-25zcr\""
Jun 02 15:47:27.214733 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.214715    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7tj9z"]
Jun 02 15:47:27.214812 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.214737    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w2q49"]
Jun 02 15:47:27.214864 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.214844    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:27.218689 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.218672    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 02 15:47:27.218964 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.218951    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 02 15:47:27.219230 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.219215    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 02 15:47:27.219852 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.219834    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gg4df\""
Jun 02 15:47:27.270528 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270508    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5083914-03b4-4dd1-947b-5e55f525b04d-ca-trust-extracted\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.270654 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.270654 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-certificates\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.270761 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270652    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:27.270761 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270709    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-bound-sa-token\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.270761 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270730    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-trusted-ca\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.270761 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270750    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-installation-pull-secrets\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.270968 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.270783    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:27.270968 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270792    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-image-registry-private-configuration\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.270968 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.270803    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:27.270968 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.270824    2561 projected.go:194] Error preparing data for projected volume kube-api-access-xjf4j for pod openshift-network-diagnostics/network-check-target-rmnx9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:27.270968 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.270822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sqq6\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-kube-api-access-5sqq6\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.271222 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.271113    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j podName:5467a93a-b318-4861-b91b-11becab47db8 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:59.271093673 +0000 UTC m=+66.465873154 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xjf4j" (UniqueName: "kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j") pod "network-check-target-rmnx9" (UID: "5467a93a-b318-4861-b91b-11becab47db8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:27.371487 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371405    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-bound-sa-token\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.371487 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371456    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-trusted-ca\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.371487 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371486    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-installation-pull-secrets\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.371752 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phtpk\" (UniqueName: \"kubernetes.io/projected/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-kube-api-access-phtpk\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.371752 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:27.371851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371799    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-image-registry-private-configuration\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.371911 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371854    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5sqq6\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-kube-api-access-5sqq6\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.371911 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371904    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-config-volume\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.372012 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.371936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.372012 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.372004    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5083914-03b4-4dd1-947b-5e55f525b04d-ca-trust-extracted\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.372108 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.372063    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.372108 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.372101    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-certificates\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.372241 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.372148    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-tmp-dir\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.372241 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.372210    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552gt\" (UniqueName: \"kubernetes.io/projected/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-kube-api-access-552gt\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:27.372335 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.372251    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:47:27.372335 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.372270    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659f487f9f-qbcmr: secret "image-registry-tls" not found
Jun 02 15:47:27.372335 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.372332    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls podName:a5083914-03b4-4dd1-947b-5e55f525b04d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:27.872311587 +0000 UTC m=+35.067091079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls") pod "image-registry-659f487f9f-qbcmr" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d") : secret "image-registry-tls" not found
Jun 02 15:47:27.372484 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.372384    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5083914-03b4-4dd1-947b-5e55f525b04d-ca-trust-extracted\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.372643 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.372623    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-certificates\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.372960 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.372940    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-trusted-ca\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.375739 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.375719    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-image-registry-private-configuration\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.375838 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.375739    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-installation-pull-secrets\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.382948 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.382925    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sqq6\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-kube-api-access-5sqq6\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.383993 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.383976    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-bound-sa-token\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.428156 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.428136    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:27.428253 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.428183    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:27.428253 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.428228    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:27.431657 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.431640    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m8f5r\""
Jun 02 15:47:27.431751 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.431735    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:27.431981 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.431843    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 02 15:47:27.431981 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.431914    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:27.431981 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.431927    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:27.432125 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.432045    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j9fj4\""
Jun 02 15:47:27.473089 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.473071    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-tmp-dir\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.473209 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.473108    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-552gt\" (UniqueName: \"kubernetes.io/projected/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-kube-api-access-552gt\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:27.473209 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.473139    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-phtpk\" (UniqueName: \"kubernetes.io/projected/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-kube-api-access-phtpk\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.473209 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.473178    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:27.473209 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.473206    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-config-volume\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.473365 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.473222    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.473365 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.473236    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:27.473365 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.473288    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert podName:a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc nodeName:}" failed. No retries permitted until 2026-06-02 15:47:27.973274674 +0000 UTC m=+35.168054152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert") pod "ingress-canary-w2q49" (UID: "a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc") : secret "canary-serving-cert" not found
Jun 02 15:47:27.473365 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.473304    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:27.473365 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.473361    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls podName:5dbb0410-abce-4622-bd7c-dd7ed0faa3b5 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:27.973344842 +0000 UTC m=+35.168124321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls") pod "dns-default-7tj9z" (UID: "5dbb0410-abce-4622-bd7c-dd7ed0faa3b5") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:27.473544 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.473527    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-tmp-dir\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.473704 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.473688    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-config-volume\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.485707 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.485688    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phtpk\" (UniqueName: \"kubernetes.io/projected/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-kube-api-access-phtpk\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.485779 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.485751    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-552gt\" (UniqueName: \"kubernetes.io/projected/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-kube-api-access-552gt\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:27.876664 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.876606    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:27.876819 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.876770    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:47:27.876819 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.876793    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659f487f9f-qbcmr: secret "image-registry-tls" not found
Jun 02 15:47:27.876909 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.876848    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls podName:a5083914-03b4-4dd1-947b-5e55f525b04d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:28.876833938 +0000 UTC m=+36.071613416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls") pod "image-registry-659f487f9f-qbcmr" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d") : secret "image-registry-tls" not found
Jun 02 15:47:27.977376 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.977343    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:27.977511 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:27.977384    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:27.977511 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.977478    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:27.977511 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.977480    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:27.977619 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.977526    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls podName:5dbb0410-abce-4622-bd7c-dd7ed0faa3b5 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:28.977513366 +0000 UTC m=+36.172292844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls") pod "dns-default-7tj9z" (UID: "5dbb0410-abce-4622-bd7c-dd7ed0faa3b5") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:27.977619 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:27.977539    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert podName:a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc nodeName:}" failed. No retries permitted until 2026-06-02 15:47:28.977533111 +0000 UTC m=+36.172312589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert") pod "ingress-canary-w2q49" (UID: "a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc") : secret "canary-serving-cert" not found
Jun 02 15:47:28.684265 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:28.684230    2561 generic.go:358] "Generic (PLEG): container finished" podID="20a616f5-634d-4520-9fc3-5fe20beac101" containerID="afe0e2c1f3c3fa9f5995d2a57cdf24d0fd16cd417c1789b3e4c90666aa7a95ab" exitCode=0
Jun 02 15:47:28.684810 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:28.684289    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8htr9" event={"ID":"20a616f5-634d-4520-9fc3-5fe20beac101","Type":"ContainerDied","Data":"afe0e2c1f3c3fa9f5995d2a57cdf24d0fd16cd417c1789b3e4c90666aa7a95ab"}
Jun 02 15:47:28.884575 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:28.884549    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:28.884702 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:28.884684    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:47:28.884746 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:28.884704    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659f487f9f-qbcmr: secret "image-registry-tls" not found
Jun 02 15:47:28.884776 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:28.884756    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls podName:a5083914-03b4-4dd1-947b-5e55f525b04d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:30.884740323 +0000 UTC m=+38.079519807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls") pod "image-registry-659f487f9f-qbcmr" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d") : secret "image-registry-tls" not found
Jun 02 15:47:28.985838 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:28.985780    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:28.985838 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:28.985814    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:28.985971 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:28.985917    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:28.985971 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:28.985961    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls podName:5dbb0410-abce-4622-bd7c-dd7ed0faa3b5 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:30.985948506 +0000 UTC m=+38.180727983 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls") pod "dns-default-7tj9z" (UID: "5dbb0410-abce-4622-bd7c-dd7ed0faa3b5") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:28.986047 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:28.985919    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:28.986047 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:28.986036    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert podName:a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc nodeName:}" failed. No retries permitted until 2026-06-02 15:47:30.986024062 +0000 UTC m=+38.180803544 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert") pod "ingress-canary-w2q49" (UID: "a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc") : secret "canary-serving-cert" not found
Jun 02 15:47:29.688238 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:29.688209    2561 generic.go:358] "Generic (PLEG): container finished" podID="20a616f5-634d-4520-9fc3-5fe20beac101" containerID="54d4368baaf8a2b38c9e03b545a84939f027568b2c17a78be31b56570f158357" exitCode=0
Jun 02 15:47:29.688570 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:29.688259    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8htr9" event={"ID":"20a616f5-634d-4520-9fc3-5fe20beac101","Type":"ContainerDied","Data":"54d4368baaf8a2b38c9e03b545a84939f027568b2c17a78be31b56570f158357"}
Jun 02 15:47:30.692844 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:30.692593    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8htr9" event={"ID":"20a616f5-634d-4520-9fc3-5fe20beac101","Type":"ContainerStarted","Data":"19af121262a70c05950257c12862d4ac10dd187bec8971caca94dfa8b8020e03"}
Jun 02 15:47:30.736974 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:30.736929    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8htr9" podStartSLOduration=6.024606464 podStartE2EDuration="37.736914542s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:46:55.8973653 +0000 UTC m=+3.092144782" lastFinishedPulling="2026-06-02 15:47:27.609673374 +0000 UTC m=+34.804452860" observedRunningTime="2026-06-02 15:47:30.735227158 +0000 UTC m=+37.930006658" watchObservedRunningTime="2026-06-02 15:47:30.736914542 +0000 UTC m=+37.931694071"
Jun 02 15:47:30.899872 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:30.899837    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:30.900032 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:30.899972    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:47:30.900032 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:30.899988    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659f487f9f-qbcmr: secret "image-registry-tls" not found
Jun 02 15:47:30.900097 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:30.900035    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls podName:a5083914-03b4-4dd1-947b-5e55f525b04d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:34.90002237 +0000 UTC m=+42.094801853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls") pod "image-registry-659f487f9f-qbcmr" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d") : secret "image-registry-tls" not found
Jun 02 15:47:31.000802 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:31.000777    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:31.000894 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:31.000810    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:31.000931 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:31.000908    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:31.000931 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:31.000910    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:31.000991 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:31.000952    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls podName:5dbb0410-abce-4622-bd7c-dd7ed0faa3b5 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:35.00093948 +0000 UTC m=+42.195718978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls") pod "dns-default-7tj9z" (UID: "5dbb0410-abce-4622-bd7c-dd7ed0faa3b5") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:31.000991 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:31.000964    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert podName:a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc nodeName:}" failed. No retries permitted until 2026-06-02 15:47:35.000958602 +0000 UTC m=+42.195738079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert") pod "ingress-canary-w2q49" (UID: "a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc") : secret "canary-serving-cert" not found
Jun 02 15:47:34.930517 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:34.930482    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:34.930987 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:34.930650    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:47:34.930987 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:34.930674    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659f487f9f-qbcmr: secret "image-registry-tls" not found
Jun 02 15:47:34.930987 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:34.930746    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls podName:a5083914-03b4-4dd1-947b-5e55f525b04d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:42.930724007 +0000 UTC m=+50.125503501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls") pod "image-registry-659f487f9f-qbcmr" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d") : secret "image-registry-tls" not found
Jun 02 15:47:35.031677 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:35.031643    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:35.031677 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:35.031682    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:35.031909 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:35.031792    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:35.031909 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:35.031804    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:35.031909 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:35.031855    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert podName:a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc nodeName:}" failed. No retries permitted until 2026-06-02 15:47:43.031840681 +0000 UTC m=+50.226620159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert") pod "ingress-canary-w2q49" (UID: "a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc") : secret "canary-serving-cert" not found
Jun 02 15:47:35.031909 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:35.031869    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls podName:5dbb0410-abce-4622-bd7c-dd7ed0faa3b5 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:43.031862907 +0000 UTC m=+50.226642385 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls") pod "dns-default-7tj9z" (UID: "5dbb0410-abce-4622-bd7c-dd7ed0faa3b5") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:42.990670 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:42.990629    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:42.991263 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:42.990801    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:47:42.991263 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:42.990832    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659f487f9f-qbcmr: secret "image-registry-tls" not found
Jun 02 15:47:42.991263 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:42.990905    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls podName:a5083914-03b4-4dd1-947b-5e55f525b04d nodeName:}" failed. No retries permitted until 2026-06-02 15:47:58.990885361 +0000 UTC m=+66.185664865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls") pod "image-registry-659f487f9f-qbcmr" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d") : secret "image-registry-tls" not found
Jun 02 15:47:43.091755 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:43.091723    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:43.091951 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:43.091764    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:43.091951 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:43.091850    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:43.091951 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:43.091856    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:43.091951 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:43.091897    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls podName:5dbb0410-abce-4622-bd7c-dd7ed0faa3b5 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:59.091884294 +0000 UTC m=+66.286663772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls") pod "dns-default-7tj9z" (UID: "5dbb0410-abce-4622-bd7c-dd7ed0faa3b5") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:43.091951 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:43.091910    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert podName:a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc nodeName:}" failed. No retries permitted until 2026-06-02 15:47:59.091904293 +0000 UTC m=+66.286683771 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert") pod "ingress-canary-w2q49" (UID: "a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc") : secret "canary-serving-cert" not found
Jun 02 15:47:59.003294 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.003256    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:47:59.003741 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.003402    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:47:59.003741 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.003419    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659f487f9f-qbcmr: secret "image-registry-tls" not found
Jun 02 15:47:59.003741 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.003482    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls podName:a5083914-03b4-4dd1-947b-5e55f525b04d nodeName:}" failed. No retries permitted until 2026-06-02 15:48:31.003465666 +0000 UTC m=+98.198245144 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls") pod "image-registry-659f487f9f-qbcmr" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d") : secret "image-registry-tls" not found
Jun 02 15:47:59.103537 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.103493    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:47:59.103537 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.103547    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:47:59.103737 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.103636    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:59.103737 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.103638    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:59.103737 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.103690    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls podName:5dbb0410-abce-4622-bd7c-dd7ed0faa3b5 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:31.103676276 +0000 UTC m=+98.298455754 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls") pod "dns-default-7tj9z" (UID: "5dbb0410-abce-4622-bd7c-dd7ed0faa3b5") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:59.103832 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.103743    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:47:59.103832 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.103788    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert podName:a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc nodeName:}" failed. No retries permitted until 2026-06-02 15:48:31.10377348 +0000 UTC m=+98.298552958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert") pod "ingress-canary-w2q49" (UID: "a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc") : secret "canary-serving-cert" not found
Jun 02 15:47:59.107439 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.107421    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:59.114348 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.114329    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:47:59.114443 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:47:59.114393    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:49:03.114377707 +0000 UTC m=+130.309157201 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : secret "metrics-daemon-secret" not found
Jun 02 15:47:59.204927 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.204893    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:59.211131 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.211109    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 02 15:47:59.218307 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.218290    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f77022b6-028e-4650-b07e-48525a6590f2-original-pull-secret\") pod \"global-pull-secret-syncer-pn7nc\" (UID: \"f77022b6-028e-4650-b07e-48525a6590f2\") " pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:59.244636 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.244617    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pn7nc"
Jun 02 15:47:59.306030 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.305951    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:59.309604 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.309586    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:59.319378 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.319356    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:59.330495 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.330471    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjf4j\" (UniqueName: \"kubernetes.io/projected/5467a93a-b318-4861-b91b-11becab47db8-kube-api-access-xjf4j\") pod \"network-check-target-rmnx9\" (UID: \"5467a93a-b318-4861-b91b-11becab47db8\") " pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:59.407369 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.407341    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pn7nc"]
Jun 02 15:47:59.412205 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:47:59.412173    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77022b6_028e_4650_b07e_48525a6590f2.slice/crio-9705052ce62e313fdb9b492f829fbe17728f1d3ea9159b8bd82117cbc7c8e081 WatchSource:0}: Error finding container 9705052ce62e313fdb9b492f829fbe17728f1d3ea9159b8bd82117cbc7c8e081: Status 404 returned error can't find the container with id 9705052ce62e313fdb9b492f829fbe17728f1d3ea9159b8bd82117cbc7c8e081
Jun 02 15:47:59.541762 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.541734    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-m8f5r\""
Jun 02 15:47:59.549500 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.549472    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:47:59.687175 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.687129    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rmnx9"]
Jun 02 15:47:59.690945 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:47:59.690918    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5467a93a_b318_4861_b91b_11becab47db8.slice/crio-c776b27456d608422fdfd74b7589f5141a10a506eb52d535b0d75bf7ed7b8f70 WatchSource:0}: Error finding container c776b27456d608422fdfd74b7589f5141a10a506eb52d535b0d75bf7ed7b8f70: Status 404 returned error can't find the container with id c776b27456d608422fdfd74b7589f5141a10a506eb52d535b0d75bf7ed7b8f70
Jun 02 15:47:59.746902 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.746872    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rmnx9" event={"ID":"5467a93a-b318-4861-b91b-11becab47db8","Type":"ContainerStarted","Data":"c776b27456d608422fdfd74b7589f5141a10a506eb52d535b0d75bf7ed7b8f70"}
Jun 02 15:47:59.747845 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:47:59.747824    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pn7nc" event={"ID":"f77022b6-028e-4650-b07e-48525a6590f2","Type":"ContainerStarted","Data":"9705052ce62e313fdb9b492f829fbe17728f1d3ea9159b8bd82117cbc7c8e081"}
Jun 02 15:48:04.759320 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:04.759280    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rmnx9" event={"ID":"5467a93a-b318-4861-b91b-11becab47db8","Type":"ContainerStarted","Data":"543c5bab7a333f9a4f8cea75908fef08505bde19fc672d4ee80a8a37b7ea8baf"}
Jun 02 15:48:04.759828 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:04.759411    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:48:04.760514 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:04.760492    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pn7nc" event={"ID":"f77022b6-028e-4650-b07e-48525a6590f2","Type":"ContainerStarted","Data":"be012c603b7f8310e8fa2a809d09849a29dd9143f74bbd70a414ec0cbe9b855a"}
Jun 02 15:48:04.778538 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:04.778502    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rmnx9" podStartSLOduration=67.429973678 podStartE2EDuration="1m11.778491537s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:47:59.692835324 +0000 UTC m=+66.887614802" lastFinishedPulling="2026-06-02 15:48:04.041353164 +0000 UTC m=+71.236132661" observedRunningTime="2026-06-02 15:48:04.778189201 +0000 UTC m=+71.972968700" watchObservedRunningTime="2026-06-02 15:48:04.778491537 +0000 UTC m=+71.973271036"
Jun 02 15:48:04.798461 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:04.798418    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pn7nc" podStartSLOduration=67.168142795 podStartE2EDuration="1m11.798407308s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:47:59.413713486 +0000 UTC m=+66.608492964" lastFinishedPulling="2026-06-02 15:48:04.043977996 +0000 UTC m=+71.238757477" observedRunningTime="2026-06-02 15:48:04.797941881 +0000 UTC m=+71.992721379" watchObservedRunningTime="2026-06-02 15:48:04.798407308 +0000 UTC m=+71.993186808"
Jun 02 15:48:31.030353 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:31.030313    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:48:31.030719 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:31.030436    2561 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 02 15:48:31.030719 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:31.030448    2561 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-659f487f9f-qbcmr: secret "image-registry-tls" not found
Jun 02 15:48:31.030719 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:31.030499    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls podName:a5083914-03b4-4dd1-947b-5e55f525b04d nodeName:}" failed. No retries permitted until 2026-06-02 15:49:35.030484999 +0000 UTC m=+162.225264478 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls") pod "image-registry-659f487f9f-qbcmr" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d") : secret "image-registry-tls" not found
Jun 02 15:48:31.130789 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:31.130758    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:48:31.130885 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:31.130806    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:48:31.130928 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:31.130891    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:48:31.130928 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:31.130894    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:48:31.130999 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:31.130934    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls podName:5dbb0410-abce-4622-bd7c-dd7ed0faa3b5 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:35.130922695 +0000 UTC m=+162.325702173 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls") pod "dns-default-7tj9z" (UID: "5dbb0410-abce-4622-bd7c-dd7ed0faa3b5") : secret "dns-default-metrics-tls" not found
Jun 02 15:48:31.130999 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:31.130962    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert podName:a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc nodeName:}" failed. No retries permitted until 2026-06-02 15:49:35.130947805 +0000 UTC m=+162.325727283 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert") pod "ingress-canary-w2q49" (UID: "a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc") : secret "canary-serving-cert" not found
Jun 02 15:48:35.764569 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:35.764539    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rmnx9"
Jun 02 15:48:49.843759 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.843726    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-86d7c6cc77-c4r5l"]
Jun 02 15:48:49.846490 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.846470    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:49.853838 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.853816    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jun 02 15:48:49.854207 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.854193    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jun 02 15:48:49.854314 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.854300    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vsdjc\""
Jun 02 15:48:49.854354 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.854338    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:49.854866 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.854840    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jun 02 15:48:49.860791 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.860774    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jun 02 15:48:49.866602 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.866585    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-86d7c6cc77-c4r5l"]
Jun 02 15:48:49.952129 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.952105    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"]
Jun 02 15:48:49.953877 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.953863    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:49.957004 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.956982    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jun 02 15:48:49.957310 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.957297    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jun 02 15:48:49.957471 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.957459    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kwds7\""
Jun 02 15:48:49.958137 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.958121    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:49.958706 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.958686    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jun 02 15:48:49.961381 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.961233    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-674954b4d5-mrfmn"]
Jun 02 15:48:49.963363 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.963336    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b8caa37-855d-4680-9fff-34dc85906216-trusted-ca\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:49.963456 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.963388    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b8caa37-855d-4680-9fff-34dc85906216-serving-cert\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:49.963456 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.963429    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n75d\" (UniqueName: \"kubernetes.io/projected/7b8caa37-855d-4680-9fff-34dc85906216-kube-api-access-8n75d\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:49.963562 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.963546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b8caa37-855d-4680-9fff-34dc85906216-config\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:49.963766 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.963742    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-788bf58df5-lv6l8"]
Jun 02 15:48:49.963947 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.963925    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:49.966066 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.966050    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:49.968925 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.968908    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jun 02 15:48:49.969040 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.968913    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jun 02 15:48:49.969225 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.969203    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jun 02 15:48:49.969225 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.969205    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jun 02 15:48:49.969370 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.969352    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lmcnm\""
Jun 02 15:48:49.969688 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.969672    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jun 02 15:48:49.969770 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.969700    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8nqg5\""
Jun 02 15:48:49.969770 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.969677    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jun 02 15:48:49.969883 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.969792    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 02 15:48:49.970042 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.969893    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jun 02 15:48:49.970105 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.970093    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jun 02 15:48:49.971918 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.971898    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 02 15:48:49.976644 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.976626    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jun 02 15:48:49.981362 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.981346    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"]
Jun 02 15:48:49.982387 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.982371    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-788bf58df5-lv6l8"]
Jun 02 15:48:49.983078 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:49.983062    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-674954b4d5-mrfmn"]
Jun 02 15:48:50.044969 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.044941    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"]
Jun 02 15:48:50.046771 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.046755    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.050701 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.050683    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jun 02 15:48:50.050886 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.050861    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ldnr9\""
Jun 02 15:48:50.050886 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.050879    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jun 02 15:48:50.051045 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.050928    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:50.051156 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.051142    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jun 02 15:48:50.057003 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.056986    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"]
Jun 02 15:48:50.059043 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.059029    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:50.062909 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.062890    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jun 02 15:48:50.063053 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.063035    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hrjp7\""
Jun 02 15:48:50.063131 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.063052    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jun 02 15:48:50.063200 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.063146    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:48:50.064118 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064098    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-stats-auth\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.064234 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064135    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b8caa37-855d-4680-9fff-34dc85906216-config\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.064234 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.064234 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064226    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b8caa37-855d-4680-9fff-34dc85906216-serving-cert\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.064376 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064286    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8n75d\" (UniqueName: \"kubernetes.io/projected/7b8caa37-855d-4680-9fff-34dc85906216-kube-api-access-8n75d\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.064376 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064355    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.064575 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x858d\" (UniqueName: \"kubernetes.io/projected/8ca99b2d-46f2-454b-89eb-df55517239d0-kube-api-access-x858d\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.064575 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064464    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca99b2d-46f2-454b-89eb-df55517239d0-service-ca-bundle\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.064575 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a46d01f-06de-4eaf-9762-c1c02fd68d11-serving-cert\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.064575 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca99b2d-46f2-454b-89eb-df55517239d0-serving-cert\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.064833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064614    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b8caa37-855d-4680-9fff-34dc85906216-trusted-ca\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.064833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8ca99b2d-46f2-454b-89eb-df55517239d0-snapshots\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.064833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca99b2d-46f2-454b-89eb-df55517239d0-trusted-ca-bundle\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.064833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064693    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a46d01f-06de-4eaf-9762-c1c02fd68d11-config\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.064833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gfz8\" (UniqueName: \"kubernetes.io/projected/906f54a9-b574-4bd6-b40d-e4383642b3ab-kube-api-access-9gfz8\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.065096 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.064871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca99b2d-46f2-454b-89eb-df55517239d0-tmp\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.065096 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.065000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8f9\" (UniqueName: \"kubernetes.io/projected/0a46d01f-06de-4eaf-9762-c1c02fd68d11-kube-api-access-9c8f9\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.065096 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.065026    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7b8caa37-855d-4680-9fff-34dc85906216-config\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.065096 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.065089    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-default-certificate\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.065911 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.065593    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7b8caa37-855d-4680-9fff-34dc85906216-trusted-ca\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.066866 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.066835    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7b8caa37-855d-4680-9fff-34dc85906216-serving-cert\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.067530 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.067511    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"]
Jun 02 15:48:50.073579 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.073560    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"]
Jun 02 15:48:50.081761 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.081740    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n75d\" (UniqueName: \"kubernetes.io/projected/7b8caa37-855d-4680-9fff-34dc85906216-kube-api-access-8n75d\") pod \"console-operator-86d7c6cc77-c4r5l\" (UID: \"7b8caa37-855d-4680-9fff-34dc85906216\") " pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.154081 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.154064    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:48:50.165895 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.165869    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfjj4\" (UniqueName: \"kubernetes.io/projected/1c81dd7d-ce90-4fd5-8b58-577f37f69105-kube-api-access-xfjj4\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:50.166015 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.165911    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.166015 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.165930    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x858d\" (UniqueName: \"kubernetes.io/projected/8ca99b2d-46f2-454b-89eb-df55517239d0-kube-api-access-x858d\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.166133 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166020    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca99b2d-46f2-454b-89eb-df55517239d0-service-ca-bundle\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.166133 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166067    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f59933-6166-4e8c-b115-ac029db80096-serving-cert\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.166259 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.166206    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:50.166259 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166215    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a46d01f-06de-4eaf-9762-c1c02fd68d11-serving-cert\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.166357 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.166267    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:48:50.666246772 +0000 UTC m=+117.861026251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : secret "router-metrics-certs-default" not found
Jun 02 15:48:50.166357 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64f59933-6166-4e8c-b115-ac029db80096-config\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.166471 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166385    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca99b2d-46f2-454b-89eb-df55517239d0-serving-cert\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.166504 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166482    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8ca99b2d-46f2-454b-89eb-df55517239d0-snapshots\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.166555 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166537    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca99b2d-46f2-454b-89eb-df55517239d0-trusted-ca-bundle\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.166605 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166570    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a46d01f-06de-4eaf-9762-c1c02fd68d11-config\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.166605 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166579    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca99b2d-46f2-454b-89eb-df55517239d0-service-ca-bundle\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.166704 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166625    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9gfz8\" (UniqueName: \"kubernetes.io/projected/906f54a9-b574-4bd6-b40d-e4383642b3ab-kube-api-access-9gfz8\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.166704 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166642    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca99b2d-46f2-454b-89eb-df55517239d0-tmp\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.166704 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzjv\" (UniqueName: \"kubernetes.io/projected/64f59933-6166-4e8c-b115-ac029db80096-kube-api-access-zjzjv\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.166827 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166717    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:50.166827 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166782    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8f9\" (UniqueName: \"kubernetes.io/projected/0a46d01f-06de-4eaf-9762-c1c02fd68d11-kube-api-access-9c8f9\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.166954 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166860    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-default-certificate\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.166954 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166897    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-stats-auth\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.167045 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.166961    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.167141 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.167124    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:48:50.667107305 +0000 UTC m=+117.861886798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:50.167249 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.167185    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca99b2d-46f2-454b-89eb-df55517239d0-tmp\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.167249 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.167211    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8ca99b2d-46f2-454b-89eb-df55517239d0-snapshots\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.167249 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.167214    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a46d01f-06de-4eaf-9762-c1c02fd68d11-config\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.167584 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.167553    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8ca99b2d-46f2-454b-89eb-df55517239d0-trusted-ca-bundle\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.168836 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.168815    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ca99b2d-46f2-454b-89eb-df55517239d0-serving-cert\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.169009 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.168996    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a46d01f-06de-4eaf-9762-c1c02fd68d11-serving-cert\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.169566 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.169545    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-stats-auth\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.169814 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.169797    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-default-certificate\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.178683 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.178654    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x858d\" (UniqueName: \"kubernetes.io/projected/8ca99b2d-46f2-454b-89eb-df55517239d0-kube-api-access-x858d\") pod \"insights-operator-788bf58df5-lv6l8\" (UID: \"8ca99b2d-46f2-454b-89eb-df55517239d0\") " pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.178948 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.178931    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8f9\" (UniqueName: \"kubernetes.io/projected/0a46d01f-06de-4eaf-9762-c1c02fd68d11-kube-api-access-9c8f9\") pod \"service-ca-operator-69877444c-s8fhp\" (UID: \"0a46d01f-06de-4eaf-9762-c1c02fd68d11\") " pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.179127 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.179111    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gfz8\" (UniqueName: \"kubernetes.io/projected/906f54a9-b574-4bd6-b40d-e4383642b3ab-kube-api-access-9gfz8\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.264684 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.264653    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"
Jun 02 15:48:50.267504 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.267485    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xfjj4\" (UniqueName: \"kubernetes.io/projected/1c81dd7d-ce90-4fd5-8b58-577f37f69105-kube-api-access-xfjj4\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:50.267561 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.267550    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f59933-6166-4e8c-b115-ac029db80096-serving-cert\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.267603 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.267569    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64f59933-6166-4e8c-b115-ac029db80096-config\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.267644 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.267628    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzjv\" (UniqueName: \"kubernetes.io/projected/64f59933-6166-4e8c-b115-ac029db80096-kube-api-access-zjzjv\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.267686 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.267660    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:50.267839 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.267811    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:50.267915 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.267879    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls podName:1c81dd7d-ce90-4fd5-8b58-577f37f69105 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:50.767860922 +0000 UTC m=+117.962640402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-b9bqc" (UID: "1c81dd7d-ce90-4fd5-8b58-577f37f69105") : secret "samples-operator-tls" not found
Jun 02 15:48:50.268285 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.268256    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64f59933-6166-4e8c-b115-ac029db80096-config\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.269777 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.269761    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64f59933-6166-4e8c-b115-ac029db80096-serving-cert\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.276615 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.276566    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-86d7c6cc77-c4r5l"]
Jun 02 15:48:50.278643 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.278620    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-788bf58df5-lv6l8"
Jun 02 15:48:50.279391 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.279371    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzjv\" (UniqueName: \"kubernetes.io/projected/64f59933-6166-4e8c-b115-ac029db80096-kube-api-access-zjzjv\") pod \"kube-storage-version-migrator-operator-96ddc787c-pn6mk\" (UID: \"64f59933-6166-4e8c-b115-ac029db80096\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.279571 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.279554    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfjj4\" (UniqueName: \"kubernetes.io/projected/1c81dd7d-ce90-4fd5-8b58-577f37f69105-kube-api-access-xfjj4\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:50.280508 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:48:50.280489    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8caa37_855d_4680_9fff_34dc85906216.slice/crio-ee6a01b88b96ea2cb809cc7a9ac8870a6283cf7e9db6061216c8c5ac6c89faca WatchSource:0}: Error finding container ee6a01b88b96ea2cb809cc7a9ac8870a6283cf7e9db6061216c8c5ac6c89faca: Status 404 returned error can't find the container with id ee6a01b88b96ea2cb809cc7a9ac8870a6283cf7e9db6061216c8c5ac6c89faca
Jun 02 15:48:50.355497 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.355470    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"
Jun 02 15:48:50.400839 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.399962    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp"]
Jun 02 15:48:50.423521 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.423492    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-788bf58df5-lv6l8"]
Jun 02 15:48:50.426597 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:48:50.426573    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca99b2d_46f2_454b_89eb_df55517239d0.slice/crio-fe3a51db8de5a49375558dc59ffb47ad625a976aa628b29abd12212d7c0646c6 WatchSource:0}: Error finding container fe3a51db8de5a49375558dc59ffb47ad625a976aa628b29abd12212d7c0646c6: Status 404 returned error can't find the container with id fe3a51db8de5a49375558dc59ffb47ad625a976aa628b29abd12212d7c0646c6
Jun 02 15:48:50.485944 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.485917    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk"]
Jun 02 15:48:50.488746 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:48:50.488719    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f59933_6166_4e8c_b115_ac029db80096.slice/crio-ef68626288c9ae7eae4c25840bc3ef85734526d18080ea27f6d965fd2663fb2e WatchSource:0}: Error finding container ef68626288c9ae7eae4c25840bc3ef85734526d18080ea27f6d965fd2663fb2e: Status 404 returned error can't find the container with id ef68626288c9ae7eae4c25840bc3ef85734526d18080ea27f6d965fd2663fb2e
Jun 02 15:48:50.671444 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.671361    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.671444 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.671403    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:50.671590 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.671533    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:48:51.671515294 +0000 UTC m=+118.866294771 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:50.671590 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.671567    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:50.671671 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.671610    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:48:51.671598308 +0000 UTC m=+118.866377785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : secret "router-metrics-certs-default" not found
Jun 02 15:48:50.773034 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.772985    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:50.773716 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.773320    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:50.773716 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:50.773403    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls podName:1c81dd7d-ce90-4fd5-8b58-577f37f69105 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:51.773381683 +0000 UTC m=+118.968161165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-b9bqc" (UID: "1c81dd7d-ce90-4fd5-8b58-577f37f69105") : secret "samples-operator-tls" not found
Jun 02 15:48:50.845077 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.845033    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk" event={"ID":"64f59933-6166-4e8c-b115-ac029db80096","Type":"ContainerStarted","Data":"ef68626288c9ae7eae4c25840bc3ef85734526d18080ea27f6d965fd2663fb2e"}
Jun 02 15:48:50.846322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.846297    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp" event={"ID":"0a46d01f-06de-4eaf-9762-c1c02fd68d11","Type":"ContainerStarted","Data":"8413ae2abb8b655670a9f27f56fa36de55d63529a9b0d99369f0594cde430029"}
Jun 02 15:48:50.847492 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.847459    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l" event={"ID":"7b8caa37-855d-4680-9fff-34dc85906216","Type":"ContainerStarted","Data":"ee6a01b88b96ea2cb809cc7a9ac8870a6283cf7e9db6061216c8c5ac6c89faca"}
Jun 02 15:48:50.848853 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:50.848831    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-788bf58df5-lv6l8" event={"ID":"8ca99b2d-46f2-454b-89eb-df55517239d0","Type":"ContainerStarted","Data":"fe3a51db8de5a49375558dc59ffb47ad625a976aa628b29abd12212d7c0646c6"}
Jun 02 15:48:51.684177 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:51.684100    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:51.684177 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:51.684178    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:51.684475 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:51.684291    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:48:53.684268861 +0000 UTC m=+120.879048339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:51.684475 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:51.684360    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:51.684475 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:51.684422    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:48:53.684404996 +0000 UTC m=+120.879184483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : secret "router-metrics-certs-default" not found
Jun 02 15:48:51.785792 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:51.785370    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:51.785792 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:51.785569    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:51.785792 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:51.785640    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls podName:1c81dd7d-ce90-4fd5-8b58-577f37f69105 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:53.785620495 +0000 UTC m=+120.980399973 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-b9bqc" (UID: "1c81dd7d-ce90-4fd5-8b58-577f37f69105") : secret "samples-operator-tls" not found
Jun 02 15:48:53.703852 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.703827    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:53.704227 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.703865    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:53.704227 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:53.703964    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:53.704227 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:53.703987    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:48:57.703968488 +0000 UTC m=+124.898747978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:53.704227 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:53.704005    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:48:57.703997083 +0000 UTC m=+124.898776561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : secret "router-metrics-certs-default" not found
Jun 02 15:48:53.804336 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.804310    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:53.804475 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:53.804460    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:53.804533 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:53.804524    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls podName:1c81dd7d-ce90-4fd5-8b58-577f37f69105 nodeName:}" failed. No retries permitted until 2026-06-02 15:48:57.804507641 +0000 UTC m=+124.999287120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-b9bqc" (UID: "1c81dd7d-ce90-4fd5-8b58-577f37f69105") : secret "samples-operator-tls" not found
Jun 02 15:48:53.857938 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.857905    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-788bf58df5-lv6l8" event={"ID":"8ca99b2d-46f2-454b-89eb-df55517239d0","Type":"ContainerStarted","Data":"5f012ee8d253147d23b3ba45fa876403ea6bb08bfdc9943187382d3aab891c9c"}
Jun 02 15:48:53.859273 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.859234    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk" event={"ID":"64f59933-6166-4e8c-b115-ac029db80096","Type":"ContainerStarted","Data":"c37a3584c15d190a5539599e9a2eedd0fdce97c1886641a1af5bb1a3ec933bb8"}
Jun 02 15:48:53.860501 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.860476    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp" event={"ID":"0a46d01f-06de-4eaf-9762-c1c02fd68d11","Type":"ContainerStarted","Data":"4cc09a7382a75ed535ed6decea611e76ff51134614eeca5ef10579d78d03469a"}
Jun 02 15:48:53.861915 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.861868    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/0.log"
Jun 02 15:48:53.861915 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.861905    2561 generic.go:358] "Generic (PLEG): container finished" podID="7b8caa37-855d-4680-9fff-34dc85906216" containerID="7f06496a5c1a8e880b7e8f5d6ad0fa9de38ea6a1c7d0d9e56d42bf6e11d1d046" exitCode=255
Jun 02 15:48:53.862049 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.861933    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l" event={"ID":"7b8caa37-855d-4680-9fff-34dc85906216","Type":"ContainerDied","Data":"7f06496a5c1a8e880b7e8f5d6ad0fa9de38ea6a1c7d0d9e56d42bf6e11d1d046"}
Jun 02 15:48:53.862191 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.862156    2561 scope.go:117] "RemoveContainer" containerID="7f06496a5c1a8e880b7e8f5d6ad0fa9de38ea6a1c7d0d9e56d42bf6e11d1d046"
Jun 02 15:48:53.883851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.883661    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-788bf58df5-lv6l8" podStartSLOduration=1.680095428 podStartE2EDuration="4.883644163s" podCreationTimestamp="2026-06-02 15:48:49 +0000 UTC" firstStartedPulling="2026-06-02 15:48:50.429050042 +0000 UTC m=+117.623829519" lastFinishedPulling="2026-06-02 15:48:53.632598762 +0000 UTC m=+120.827378254" observedRunningTime="2026-06-02 15:48:53.882474662 +0000 UTC m=+121.077254163" watchObservedRunningTime="2026-06-02 15:48:53.883644163 +0000 UTC m=+121.078423664"
Jun 02 15:48:53.931152 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:53.931097    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp" podStartSLOduration=1.7089789469999999 podStartE2EDuration="4.931081527s" podCreationTimestamp="2026-06-02 15:48:49 +0000 UTC" firstStartedPulling="2026-06-02 15:48:50.405366095 +0000 UTC m=+117.600145574" lastFinishedPulling="2026-06-02 15:48:53.627468671 +0000 UTC m=+120.822248154" observedRunningTime="2026-06-02 15:48:53.929140724 +0000 UTC m=+121.123920223" watchObservedRunningTime="2026-06-02 15:48:53.931081527 +0000 UTC m=+121.125861028"
Jun 02 15:48:54.865710 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:54.865675    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 15:48:54.866073 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:54.866057    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/0.log"
Jun 02 15:48:54.866130 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:54.866090    2561 generic.go:358] "Generic (PLEG): container finished" podID="7b8caa37-855d-4680-9fff-34dc85906216" containerID="b45b52a83054d8da8e5972c2222c190cf460cbb23751dcbc87a52d5410857a30" exitCode=255
Jun 02 15:48:54.866232 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:54.866205    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l" event={"ID":"7b8caa37-855d-4680-9fff-34dc85906216","Type":"ContainerDied","Data":"b45b52a83054d8da8e5972c2222c190cf460cbb23751dcbc87a52d5410857a30"}
Jun 02 15:48:54.866293 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:54.866279    2561 scope.go:117] "RemoveContainer" containerID="7f06496a5c1a8e880b7e8f5d6ad0fa9de38ea6a1c7d0d9e56d42bf6e11d1d046"
Jun 02 15:48:54.866559 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:54.866534    2561 scope.go:117] "RemoveContainer" containerID="b45b52a83054d8da8e5972c2222c190cf460cbb23751dcbc87a52d5410857a30"
Jun 02 15:48:54.866757 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:54.866731    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-86d7c6cc77-c4r5l_openshift-console-operator(7b8caa37-855d-4680-9fff-34dc85906216)\"" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l" podUID="7b8caa37-855d-4680-9fff-34dc85906216"
Jun 02 15:48:54.887091 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:54.887048    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk" podStartSLOduration=1.748827354 podStartE2EDuration="4.887034449s" podCreationTimestamp="2026-06-02 15:48:50 +0000 UTC" firstStartedPulling="2026-06-02 15:48:50.490404839 +0000 UTC m=+117.685184317" lastFinishedPulling="2026-06-02 15:48:53.628611934 +0000 UTC m=+120.823391412" observedRunningTime="2026-06-02 15:48:53.955042214 +0000 UTC m=+121.149821714" watchObservedRunningTime="2026-06-02 15:48:54.887034449 +0000 UTC m=+122.081813946"
Jun 02 15:48:55.870943 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:55.870912    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 15:48:55.871454 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:55.871289    2561 scope.go:117] "RemoveContainer" containerID="b45b52a83054d8da8e5972c2222c190cf460cbb23751dcbc87a52d5410857a30"
Jun 02 15:48:55.871514 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:55.871451    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-86d7c6cc77-c4r5l_openshift-console-operator(7b8caa37-855d-4680-9fff-34dc85906216)\"" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l" podUID="7b8caa37-855d-4680-9fff-34dc85906216"
Jun 02 15:48:57.165368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.165334    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-685df94d57-5mltc"]
Jun 02 15:48:57.168272 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.168256    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.171342 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.171320    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jun 02 15:48:57.171462 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.171373    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jun 02 15:48:57.171462 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.171438    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jun 02 15:48:57.171813 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.171799    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jun 02 15:48:57.172882 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.172869    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b8vlq\""
Jun 02 15:48:57.182239 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.182220    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-685df94d57-5mltc"]
Jun 02 15:48:57.332699 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.332665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db97g\" (UniqueName: \"kubernetes.io/projected/38cd3ef6-3da2-49ed-8e09-023402c046f2-kube-api-access-db97g\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.332891 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.332734    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/38cd3ef6-3da2-49ed-8e09-023402c046f2-signing-key\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.332891 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.332816    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/38cd3ef6-3da2-49ed-8e09-023402c046f2-signing-cabundle\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.434156 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.434082    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-db97g\" (UniqueName: \"kubernetes.io/projected/38cd3ef6-3da2-49ed-8e09-023402c046f2-kube-api-access-db97g\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.434156 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.434144    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/38cd3ef6-3da2-49ed-8e09-023402c046f2-signing-key\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.434352 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.434218    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/38cd3ef6-3da2-49ed-8e09-023402c046f2-signing-cabundle\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.434843 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.434822    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/38cd3ef6-3da2-49ed-8e09-023402c046f2-signing-cabundle\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.436534 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.436506    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/38cd3ef6-3da2-49ed-8e09-023402c046f2-signing-key\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.445183 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.445147    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db97g\" (UniqueName: \"kubernetes.io/projected/38cd3ef6-3da2-49ed-8e09-023402c046f2-kube-api-access-db97g\") pod \"service-ca-685df94d57-5mltc\" (UID: \"38cd3ef6-3da2-49ed-8e09-023402c046f2\") " pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.477262 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.477234    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-685df94d57-5mltc"
Jun 02 15:48:57.603103 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.603074    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-685df94d57-5mltc"]
Jun 02 15:48:57.606004 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:48:57.605979    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cd3ef6_3da2_49ed_8e09_023402c046f2.slice/crio-597daf5302cf48d983e812231b39dadf2e4cec0d72157c184a45f1ebbc20ba90 WatchSource:0}: Error finding container 597daf5302cf48d983e812231b39dadf2e4cec0d72157c184a45f1ebbc20ba90: Status 404 returned error can't find the container with id 597daf5302cf48d983e812231b39dadf2e4cec0d72157c184a45f1ebbc20ba90
Jun 02 15:48:57.739943 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.739262    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:57.739943 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.739328    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:48:57.739943 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:57.739485    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jun 02 15:48:57.739943 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:57.739565    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:49:05.739544786 +0000 UTC m=+132.934324267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : secret "router-metrics-certs-default" not found
Jun 02 15:48:57.740256 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:57.740013    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle podName:906f54a9-b574-4bd6-b40d-e4383642b3ab nodeName:}" failed. No retries permitted until 2026-06-02 15:49:05.739984655 +0000 UTC m=+132.934764150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle") pod "router-default-674954b4d5-mrfmn" (UID: "906f54a9-b574-4bd6-b40d-e4383642b3ab") : configmap references non-existent config key: service-ca.crt
Jun 02 15:48:57.757444 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.757419    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66lpd_1f7581f7-059f-4573-aec7-312b2de9f73c/dns-node-resolver/0.log"
Jun 02 15:48:57.839956 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.839917    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:48:57.840119 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:57.840059    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jun 02 15:48:57.840194 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:48:57.840137    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls podName:1c81dd7d-ce90-4fd5-8b58-577f37f69105 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:05.840122429 +0000 UTC m=+133.034901910 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls") pod "cluster-samples-operator-755bbf9bfb-b9bqc" (UID: "1c81dd7d-ce90-4fd5-8b58-577f37f69105") : secret "samples-operator-tls" not found
Jun 02 15:48:57.876969 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.876935    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-685df94d57-5mltc" event={"ID":"38cd3ef6-3da2-49ed-8e09-023402c046f2","Type":"ContainerStarted","Data":"5089cb89d0e04b79078906e3aa160d3f2123711c701cc0c4a3b035c296004797"}
Jun 02 15:48:57.876969 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.876971    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-685df94d57-5mltc" event={"ID":"38cd3ef6-3da2-49ed-8e09-023402c046f2","Type":"ContainerStarted","Data":"597daf5302cf48d983e812231b39dadf2e4cec0d72157c184a45f1ebbc20ba90"}
Jun 02 15:48:57.897881 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:57.897834    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-685df94d57-5mltc" podStartSLOduration=0.897818176 podStartE2EDuration="897.818176ms" podCreationTimestamp="2026-06-02 15:48:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:48:57.896208696 +0000 UTC m=+125.090988195" watchObservedRunningTime="2026-06-02 15:48:57.897818176 +0000 UTC m=+125.092597675"
Jun 02 15:48:58.954087 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:48:58.954059    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zqnmk_5948a580-09d3-4717-aa41-4ed1de678f0d/node-ca/0.log"
Jun 02 15:49:00.154963 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:00.154919    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:49:00.154963 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:00.154964    2561 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:49:00.155402 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:00.155346    2561 scope.go:117] "RemoveContainer" containerID="b45b52a83054d8da8e5972c2222c190cf460cbb23751dcbc87a52d5410857a30"
Jun 02 15:49:00.155520 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:00.155504    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-86d7c6cc77-c4r5l_openshift-console-operator(7b8caa37-855d-4680-9fff-34dc85906216)\"" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l" podUID="7b8caa37-855d-4680-9fff-34dc85906216"
Jun 02 15:49:00.355294 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:00.355267    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-96ddc787c-pn6mk_64f59933-6166-4e8c-b115-ac029db80096/kube-storage-version-migrator-operator/0.log"
Jun 02 15:49:03.184457 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:03.184423    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:49:03.184912 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:03.184580    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 02 15:49:03.184912 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:03.184665    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs podName:6d86be44-4451-4903-bf56-5179cec7698d nodeName:}" failed. No retries permitted until 2026-06-02 15:51:05.184643352 +0000 UTC m=+252.379422837 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs") pod "network-metrics-daemon-48g4z" (UID: "6d86be44-4451-4903-bf56-5179cec7698d") : secret "metrics-daemon-secret" not found
Jun 02 15:49:05.805200 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.805141    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:05.805200 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.805203    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:05.805737 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.805716    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906f54a9-b574-4bd6-b40d-e4383642b3ab-service-ca-bundle\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:05.807444 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.807419    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/906f54a9-b574-4bd6-b40d-e4383642b3ab-metrics-certs\") pod \"router-default-674954b4d5-mrfmn\" (UID: \"906f54a9-b574-4bd6-b40d-e4383642b3ab\") " pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:05.877423 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.877389    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lmcnm\""
Jun 02 15:49:05.884223 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.884193    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:05.905931 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.905903    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:49:05.908111 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.908085    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1c81dd7d-ce90-4fd5-8b58-577f37f69105-samples-operator-tls\") pod \"cluster-samples-operator-755bbf9bfb-b9bqc\" (UID: \"1c81dd7d-ce90-4fd5-8b58-577f37f69105\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:49:05.970398 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.970370    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-hrjp7\""
Jun 02 15:49:05.977707 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:05.977684    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"
Jun 02 15:49:06.010520 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:06.010489    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-674954b4d5-mrfmn"]
Jun 02 15:49:06.014313 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:06.014285    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906f54a9_b574_4bd6_b40d_e4383642b3ab.slice/crio-ba42128b596629f9235b2818b8c3d5b38ccfe8b623a7edb2826a280bcae3e436 WatchSource:0}: Error finding container ba42128b596629f9235b2818b8c3d5b38ccfe8b623a7edb2826a280bcae3e436: Status 404 returned error can't find the container with id ba42128b596629f9235b2818b8c3d5b38ccfe8b623a7edb2826a280bcae3e436
Jun 02 15:49:06.114510 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:06.114476    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc"]
Jun 02 15:49:06.900428 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:06.900364    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-674954b4d5-mrfmn" event={"ID":"906f54a9-b574-4bd6-b40d-e4383642b3ab","Type":"ContainerStarted","Data":"c0cd1245ef2e53b573788ea46d9080a87ea1bbb2dbd1d07a572bfafaee62c369"}
Jun 02 15:49:06.900428 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:06.900408    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-674954b4d5-mrfmn" event={"ID":"906f54a9-b574-4bd6-b40d-e4383642b3ab","Type":"ContainerStarted","Data":"ba42128b596629f9235b2818b8c3d5b38ccfe8b623a7edb2826a280bcae3e436"}
Jun 02 15:49:06.901630 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:06.901605    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc" event={"ID":"1c81dd7d-ce90-4fd5-8b58-577f37f69105","Type":"ContainerStarted","Data":"307164ffb33fa1d0a070f509a6d724b56e5a50375f3e5f6f63422b11628db914"}
Jun 02 15:49:06.925445 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:06.925405    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-674954b4d5-mrfmn" podStartSLOduration=17.92538866 podStartE2EDuration="17.92538866s" podCreationTimestamp="2026-06-02 15:48:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:06.923908508 +0000 UTC m=+134.118688056" watchObservedRunningTime="2026-06-02 15:49:06.92538866 +0000 UTC m=+134.120168161"
Jun 02 15:49:07.885197 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:07.885146    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:07.887677 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:07.887656    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:07.905929 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:07.905893    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc" event={"ID":"1c81dd7d-ce90-4fd5-8b58-577f37f69105","Type":"ContainerStarted","Data":"d276d974b8c2fa334f87b992a50e052704a3a7c87b8b33c8edecfcfda371d7f0"}
Jun 02 15:49:07.906330 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:07.906006    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:07.906330 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:07.906020    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc" event={"ID":"1c81dd7d-ce90-4fd5-8b58-577f37f69105","Type":"ContainerStarted","Data":"7cbf20685741a72f859f3cbcec6cbd2ad3f4728cee1467cc133222a9bc173291"}
Jun 02 15:49:07.907063 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:07.907043    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-674954b4d5-mrfmn"
Jun 02 15:49:07.980423 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:07.980375    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-755bbf9bfb-b9bqc" podStartSLOduration=16.584196064 podStartE2EDuration="17.980361722s" podCreationTimestamp="2026-06-02 15:48:50 +0000 UTC" firstStartedPulling="2026-06-02 15:49:06.152927031 +0000 UTC m=+133.347706509" lastFinishedPulling="2026-06-02 15:49:07.549092675 +0000 UTC m=+134.743872167" observedRunningTime="2026-06-02 15:49:07.979920703 +0000 UTC m=+135.174700203" watchObservedRunningTime="2026-06-02 15:49:07.980361722 +0000 UTC m=+135.175141222"
Jun 02 15:49:14.428629 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:14.428601    2561 scope.go:117] "RemoveContainer" containerID="b45b52a83054d8da8e5972c2222c190cf460cbb23751dcbc87a52d5410857a30"
Jun 02 15:49:14.924121 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:14.924092    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 15:49:14.924291 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:14.924184    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l" event={"ID":"7b8caa37-855d-4680-9fff-34dc85906216","Type":"ContainerStarted","Data":"33479e547f31ab865ccf04a21f358971995214091e0af531d166a4b546ce5e7b"}
Jun 02 15:49:14.924477 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:14.924451    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:49:14.943949 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:14.943907    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l" podStartSLOduration=22.60409833 podStartE2EDuration="25.94389639s" podCreationTimestamp="2026-06-02 15:48:49 +0000 UTC" firstStartedPulling="2026-06-02 15:48:50.284514052 +0000 UTC m=+117.479293537" lastFinishedPulling="2026-06-02 15:48:53.624312105 +0000 UTC m=+120.819091597" observedRunningTime="2026-06-02 15:49:14.943451489 +0000 UTC m=+142.138230992" watchObservedRunningTime="2026-06-02 15:49:14.94389639 +0000 UTC m=+142.138675889"
Jun 02 15:49:15.369037 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:15.368970    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-86d7c6cc77-c4r5l"
Jun 02 15:49:18.711324 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.711294    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-drq7l"]
Jun 02 15:49:18.713727 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.713704    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.719826 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.719804    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 02 15:49:18.719954 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.719830    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-589bn\""
Jun 02 15:49:18.719954 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.719842    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 02 15:49:18.727607 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.727587    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-drq7l"]
Jun 02 15:49:18.748869 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.748844    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc"]
Jun 02 15:49:18.751129 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.751107    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc"
Jun 02 15:49:18.756610 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.756583    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mn8lb\""
Jun 02 15:49:18.767878 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.767857    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc"]
Jun 02 15:49:18.813179 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.813134    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-crio-socket\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.813330 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.813199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljq46\" (UniqueName: \"kubernetes.io/projected/c18af438-7ff0-4e15-83a0-320a446f2bd3-kube-api-access-ljq46\") pod \"network-check-source-56d84b9f44-b2zdc\" (UID: \"c18af438-7ff0-4e15-83a0-320a446f2bd3\") " pod="openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc"
Jun 02 15:49:18.813330 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.813225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.813330 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.813309    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.813447 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.813353    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-data-volume\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.813447 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.813382    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx7wv\" (UniqueName: \"kubernetes.io/projected/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-kube-api-access-lx7wv\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.914623 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.914592    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lx7wv\" (UniqueName: \"kubernetes.io/projected/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-kube-api-access-lx7wv\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.914785 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.914639    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-crio-socket\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.914785 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.914669    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ljq46\" (UniqueName: \"kubernetes.io/projected/c18af438-7ff0-4e15-83a0-320a446f2bd3-kube-api-access-ljq46\") pod \"network-check-source-56d84b9f44-b2zdc\" (UID: \"c18af438-7ff0-4e15-83a0-320a446f2bd3\") " pod="openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc"
Jun 02 15:49:18.914785 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.914718    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.914946 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.914785    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-crio-socket\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.914946 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.914789    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.914946 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.914877    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-data-volume\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.915254 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.915222    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-data-volume\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.915431 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.915408    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.917124 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.917102    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.936044 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.936016    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx7wv\" (UniqueName: \"kubernetes.io/projected/5c7930c7-89b3-4575-ba9a-f8dac8c9e87f-kube-api-access-lx7wv\") pod \"insights-runtime-extractor-drq7l\" (UID: \"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f\") " pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:18.943471 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:18.943448    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljq46\" (UniqueName: \"kubernetes.io/projected/c18af438-7ff0-4e15-83a0-320a446f2bd3-kube-api-access-ljq46\") pod \"network-check-source-56d84b9f44-b2zdc\" (UID: \"c18af438-7ff0-4e15-83a0-320a446f2bd3\") " pod="openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc"
Jun 02 15:49:19.025321 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.025250    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-drq7l"
Jun 02 15:49:19.059709 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.059673    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc"
Jun 02 15:49:19.181900 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.181867    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-drq7l"]
Jun 02 15:49:19.184989 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:19.184960    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7930c7_89b3_4575_ba9a_f8dac8c9e87f.slice/crio-cde894e94dcceedda9f4791575150a22574d8c1f8ccbdce4c9304560fcee58fe WatchSource:0}: Error finding container cde894e94dcceedda9f4791575150a22574d8c1f8ccbdce4c9304560fcee58fe: Status 404 returned error can't find the container with id cde894e94dcceedda9f4791575150a22574d8c1f8ccbdce4c9304560fcee58fe
Jun 02 15:49:19.211266 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.211238    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc"]
Jun 02 15:49:19.215258 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:19.215236    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18af438_7ff0_4e15_83a0_320a446f2bd3.slice/crio-3db72c6c05e0925bb64c6f80dcb31004da5eaf6b5b647b2bab27ac2f7cb76b93 WatchSource:0}: Error finding container 3db72c6c05e0925bb64c6f80dcb31004da5eaf6b5b647b2bab27ac2f7cb76b93: Status 404 returned error can't find the container with id 3db72c6c05e0925bb64c6f80dcb31004da5eaf6b5b647b2bab27ac2f7cb76b93
Jun 02 15:49:19.939748 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.939691    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc" event={"ID":"c18af438-7ff0-4e15-83a0-320a446f2bd3","Type":"ContainerStarted","Data":"a2635cc4dbe6051ad2983f79a6e47980ad91fa04c51f1a38d2d5102f55356817"}
Jun 02 15:49:19.939748 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.939749    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc" event={"ID":"c18af438-7ff0-4e15-83a0-320a446f2bd3","Type":"ContainerStarted","Data":"3db72c6c05e0925bb64c6f80dcb31004da5eaf6b5b647b2bab27ac2f7cb76b93"}
Jun 02 15:49:19.941383 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.941362    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drq7l" event={"ID":"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f","Type":"ContainerStarted","Data":"75fc8cbb92f2f773f2323c412696576af7512884e102adbaeb31289646e9f226"}
Jun 02 15:49:19.941477 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.941388    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drq7l" event={"ID":"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f","Type":"ContainerStarted","Data":"533be31cd4e507aef5ae977a4b2f51b1524620bfe54cbf3b7b5d0b247cb312d9"}
Jun 02 15:49:19.941477 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.941397    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drq7l" event={"ID":"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f","Type":"ContainerStarted","Data":"cde894e94dcceedda9f4791575150a22574d8c1f8ccbdce4c9304560fcee58fe"}
Jun 02 15:49:19.962784 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:19.962734    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56d84b9f44-b2zdc" podStartSLOduration=1.962719126 podStartE2EDuration="1.962719126s" podCreationTimestamp="2026-06-02 15:49:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:19.961100781 +0000 UTC m=+147.155880280" watchObservedRunningTime="2026-06-02 15:49:19.962719126 +0000 UTC m=+147.157498620"
Jun 02 15:49:21.073324 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.073302    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-8444899ccb-wbzlp"]
Jun 02 15:49:21.078312 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.078292    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.082974 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.082952    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jun 02 15:49:21.084306 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.084276    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jun 02 15:49:21.084306 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.084298    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jun 02 15:49:21.084449 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.084296    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-928l8\""
Jun 02 15:49:21.084499 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.084460    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 02 15:49:21.084550 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.084506    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jun 02 15:49:21.084550 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.084537    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 02 15:49:21.084833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.084818    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jun 02 15:49:21.089359 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.089341    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8444899ccb-wbzlp"]
Jun 02 15:49:21.134684 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.134663    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-console-config\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.134753 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.134692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-oauth-serving-cert\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.134753 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.134711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmk4d\" (UniqueName: \"kubernetes.io/projected/850916db-3f77-4547-9a28-8a034d41704f-kube-api-access-wmk4d\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.134818 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.134762    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-oauth-config\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.134852 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.134834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-service-ca\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.134899 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.134888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-serving-cert\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.235883 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.235850    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-serving-cert\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.236030 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.235906    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-console-config\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.236030 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.235928    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-oauth-serving-cert\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.236030 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.235944    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wmk4d\" (UniqueName: \"kubernetes.io/projected/850916db-3f77-4547-9a28-8a034d41704f-kube-api-access-wmk4d\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.236030 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.235970    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-oauth-config\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.236030 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.236018    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-service-ca\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.236689 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.236662    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-console-config\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.236797 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.236764    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-oauth-serving-cert\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.236797 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.236764    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-service-ca\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.238131 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.238114    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-serving-cert\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.238260 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.238244    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-oauth-config\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.244969 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.244944    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmk4d\" (UniqueName: \"kubernetes.io/projected/850916db-3f77-4547-9a28-8a034d41704f-kube-api-access-wmk4d\") pod \"console-8444899ccb-wbzlp\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") " pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.387796 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.387762    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:21.507831 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.507799    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8444899ccb-wbzlp"]
Jun 02 15:49:21.510998 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:21.510968    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850916db_3f77_4547_9a28_8a034d41704f.slice/crio-013d505d262dddd485a8e05c71c1939d4b8b8c71a2a86b2bed7764d44c0577cc WatchSource:0}: Error finding container 013d505d262dddd485a8e05c71c1939d4b8b8c71a2a86b2bed7764d44c0577cc: Status 404 returned error can't find the container with id 013d505d262dddd485a8e05c71c1939d4b8b8c71a2a86b2bed7764d44c0577cc
Jun 02 15:49:21.948045 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.948013    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8444899ccb-wbzlp" event={"ID":"850916db-3f77-4547-9a28-8a034d41704f","Type":"ContainerStarted","Data":"013d505d262dddd485a8e05c71c1939d4b8b8c71a2a86b2bed7764d44c0577cc"}
Jun 02 15:49:21.949568 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.949541    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-drq7l" event={"ID":"5c7930c7-89b3-4575-ba9a-f8dac8c9e87f","Type":"ContainerStarted","Data":"a9b5f13b604687ecbc86f405859ad3e1af5df147cb26aa376753842a429e6ed1"}
Jun 02 15:49:21.981405 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:21.981363    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-drq7l" podStartSLOduration=2.151240942 podStartE2EDuration="3.981349583s" podCreationTimestamp="2026-06-02 15:49:18 +0000 UTC" firstStartedPulling="2026-06-02 15:49:19.247557063 +0000 UTC m=+146.442336548" lastFinishedPulling="2026-06-02 15:49:21.0776657 +0000 UTC m=+148.272445189" observedRunningTime="2026-06-02 15:49:21.980781766 +0000 UTC m=+149.175561266" watchObservedRunningTime="2026-06-02 15:49:21.981349583 +0000 UTC m=+149.176129083"
Jun 02 15:49:24.959448 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:24.959408    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8444899ccb-wbzlp" event={"ID":"850916db-3f77-4547-9a28-8a034d41704f","Type":"ContainerStarted","Data":"a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1"}
Jun 02 15:49:24.981031 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:24.980938    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-8444899ccb-wbzlp" podStartSLOduration=0.773978317 podStartE2EDuration="3.980925519s" podCreationTimestamp="2026-06-02 15:49:21 +0000 UTC" firstStartedPulling="2026-06-02 15:49:21.512880817 +0000 UTC m=+148.707660299" lastFinishedPulling="2026-06-02 15:49:24.71982801 +0000 UTC m=+151.914607501" observedRunningTime="2026-06-02 15:49:24.979978509 +0000 UTC m=+152.174758008" watchObservedRunningTime="2026-06-02 15:49:24.980925519 +0000 UTC m=+152.175705019"
Jun 02 15:49:27.665428 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.665396    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"]
Jun 02 15:49:27.705200 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.705150    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"]
Jun 02 15:49:27.705329 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.705275    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.709469 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.709429    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jun 02 15:49:27.709469 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.709434    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 02 15:49:27.709469 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.709438    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jun 02 15:49:27.709737 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.709484    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 02 15:49:27.709737 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.709438    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rgk4w\""
Jun 02 15:49:27.709737 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.709442    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 02 15:49:27.795661 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.795634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.795661 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.795666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-prometheus-operator-tls\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.795833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.795691    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zm77\" (UniqueName: \"kubernetes.io/projected/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-kube-api-access-4zm77\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.795833 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.795727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-metrics-client-ca\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.896961 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.896935    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-metrics-client-ca\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.897102 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.897035    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.897102 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.897055    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-prometheus-operator-tls\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.897102 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.897082    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4zm77\" (UniqueName: \"kubernetes.io/projected/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-kube-api-access-4zm77\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.897669 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.897647    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-metrics-client-ca\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.899433 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.899413    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-prometheus-operator-tls\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.899433 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.899419    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:27.906767 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:27.906737    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zm77\" (UniqueName: \"kubernetes.io/projected/4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c-kube-api-access-4zm77\") pod \"prometheus-operator-658bcd8bdf-sqwbg\" (UID: \"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c\") " pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:28.014459 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:28.014390    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"
Jun 02 15:49:28.136308 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:28.136266    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg"]
Jun 02 15:49:28.138689 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:28.138660    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8e2eb9_c80f_487e_a35b_7d76dd4ab47c.slice/crio-c5de56b1a3c15689257e049baecc0920e714f9c0bf028de13c8e29e4b87ee406 WatchSource:0}: Error finding container c5de56b1a3c15689257e049baecc0920e714f9c0bf028de13c8e29e4b87ee406: Status 404 returned error can't find the container with id c5de56b1a3c15689257e049baecc0920e714f9c0bf028de13c8e29e4b87ee406
Jun 02 15:49:28.970963 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:28.970933    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg" event={"ID":"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c","Type":"ContainerStarted","Data":"c5de56b1a3c15689257e049baecc0920e714f9c0bf028de13c8e29e4b87ee406"}
Jun 02 15:49:29.975035 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:29.975003    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg" event={"ID":"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c","Type":"ContainerStarted","Data":"c37087e16e8c8b95903ecc2808e53fee752c321e096f4e618f5acb7c7abf6d84"}
Jun 02 15:49:29.975035 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:29.975036    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg" event={"ID":"4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c","Type":"ContainerStarted","Data":"fcaa2959df040fc043e1cf9e4b6b171096a8e84ec11dfa6cd539a9c9aa808a61"}
Jun 02 15:49:29.998575 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:29.998530    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-658bcd8bdf-sqwbg" podStartSLOduration=1.7742416250000002 podStartE2EDuration="2.998516761s" podCreationTimestamp="2026-06-02 15:49:27 +0000 UTC" firstStartedPulling="2026-06-02 15:49:28.140626138 +0000 UTC m=+155.335405617" lastFinishedPulling="2026-06-02 15:49:29.364901276 +0000 UTC m=+156.559680753" observedRunningTime="2026-06-02 15:49:29.997467687 +0000 UTC m=+157.192247189" watchObservedRunningTime="2026-06-02 15:49:29.998516761 +0000 UTC m=+157.193296261"
Jun 02 15:49:30.195212 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:30.195151    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr" podUID="a5083914-03b4-4dd1-947b-5e55f525b04d"
Jun 02 15:49:30.209317 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:30.209279    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-7tj9z" podUID="5dbb0410-abce-4622-bd7c-dd7ed0faa3b5"
Jun 02 15:49:30.222607 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:30.222584    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-w2q49" podUID="a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc"
Jun 02 15:49:30.448586 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:30.448552    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-48g4z" podUID="6d86be44-4451-4903-bf56-5179cec7698d"
Jun 02 15:49:30.977817 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:30.977789    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:49:30.978114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:30.977789    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:49:31.388672 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:31.388646    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:31.388672 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:31.388678    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:49:31.390100 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:31.390074    2561 patch_prober.go:28] interesting pod/console-8444899ccb-wbzlp container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.18:8443/health\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body=
Jun 02 15:49:31.390225 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:31.390127    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-8444899ccb-wbzlp" podUID="850916db-3f77-4547-9a28-8a034d41704f" containerName="console" probeResult="failure" output="Get \"https://10.132.0.18:8443/health\": dial tcp 10.132.0.18:8443: connect: connection refused"
Jun 02 15:49:32.069847 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.069809    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zmpcb"]
Jun 02 15:49:32.073114 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.073094    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.076645 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.076623    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 02 15:49:32.076806 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.076785    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2wjpl\""
Jun 02 15:49:32.080485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.080464    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 02 15:49:32.080761 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.080746    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 02 15:49:32.234030 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.233993    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zffkv\" (UniqueName: \"kubernetes.io/projected/a1460ceb-a365-4964-871a-46ed94304692-kube-api-access-zffkv\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.234221 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.234086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.234221 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.234119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-root\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.234221 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.234181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-accelerators-collector-config\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.234407 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.234235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-textfile\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.234407 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.234276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-sys\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.234407 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.234300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1460ceb-a365-4964-871a-46ed94304692-metrics-client-ca\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.234407 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.234345    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-wtmp\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.234407 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.234386    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-tls\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335097 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335019    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-sys\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335097 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335054    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1460ceb-a365-4964-871a-46ed94304692-metrics-client-ca\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335097 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335083    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-wtmp\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335110    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-tls\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335124    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-sys\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335151    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zffkv\" (UniqueName: \"kubernetes.io/projected/a1460ceb-a365-4964-871a-46ed94304692-kube-api-access-zffkv\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335218    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335240    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-root\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335255    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-wtmp\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335306    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1460ceb-a365-4964-871a-46ed94304692-root\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335309    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-accelerators-collector-config\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335606 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335353    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-textfile\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335661 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335622    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-textfile\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335791 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335770    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1460ceb-a365-4964-871a-46ed94304692-metrics-client-ca\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.335856 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.335791    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-accelerators-collector-config\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.337518 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.337494    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.337518 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.337511    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1460ceb-a365-4964-871a-46ed94304692-node-exporter-tls\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.345454 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.345429    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zffkv\" (UniqueName: \"kubernetes.io/projected/a1460ceb-a365-4964-871a-46ed94304692-kube-api-access-zffkv\") pod \"node-exporter-zmpcb\" (UID: \"a1460ceb-a365-4964-871a-46ed94304692\") " pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.385260 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.385231    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zmpcb"
Jun 02 15:49:32.393607 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:32.393580    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1460ceb_a365_4964_871a_46ed94304692.slice/crio-266e69603434c49139d0c4e79f4dc829dd188a8583ab74128d1550c5e2450171 WatchSource:0}: Error finding container 266e69603434c49139d0c4e79f4dc829dd188a8583ab74128d1550c5e2450171: Status 404 returned error can't find the container with id 266e69603434c49139d0c4e79f4dc829dd188a8583ab74128d1550c5e2450171
Jun 02 15:49:32.985332 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:32.985293    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmpcb" event={"ID":"a1460ceb-a365-4964-871a-46ed94304692","Type":"ContainerStarted","Data":"266e69603434c49139d0c4e79f4dc829dd188a8583ab74128d1550c5e2450171"}
Jun 02 15:49:33.120211 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.119735    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:33.123780 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.123758    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.126872 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.126854    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jun 02 15:49:33.127808 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.127774    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jun 02 15:49:33.127928 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.127836    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jun 02 15:49:33.127928 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.127835    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jun 02 15:49:33.128236 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.128220    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jun 02 15:49:33.128334 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.128244    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jun 02 15:49:33.128883 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.128863    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jun 02 15:49:33.128942 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.128900    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5gg9r\""
Jun 02 15:49:33.129432 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.129415    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jun 02 15:49:33.157763 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.157730    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:33.158431 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.158406    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jun 02 15:49:33.243662 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.243662 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243629    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51232482-b19e-4fe8-ac02-2acaf7375f79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.243835 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243727    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51232482-b19e-4fe8-ac02-2acaf7375f79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.243835 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243757    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.243835 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-web-config\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.243835 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctxk\" (UniqueName: \"kubernetes.io/projected/51232482-b19e-4fe8-ac02-2acaf7375f79-kube-api-access-sctxk\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.244024 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.244024 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243924    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51232482-b19e-4fe8-ac02-2acaf7375f79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.244024 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.243971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.244024 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.244010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51232482-b19e-4fe8-ac02-2acaf7375f79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.244253 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.244045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.244253 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.244083    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-config-volume\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.244253 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.244230    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51232482-b19e-4fe8-ac02-2acaf7375f79-config-out\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.345621 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345585    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51232482-b19e-4fe8-ac02-2acaf7375f79-config-out\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.345792 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345637    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.345792 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345656    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51232482-b19e-4fe8-ac02-2acaf7375f79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.345792 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345686    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51232482-b19e-4fe8-ac02-2acaf7375f79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.345792 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345706    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.345792 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345734    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-web-config\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.345792 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345762    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sctxk\" (UniqueName: \"kubernetes.io/projected/51232482-b19e-4fe8-ac02-2acaf7375f79-kube-api-access-sctxk\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.346085 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345797    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.346085 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345834    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51232482-b19e-4fe8-ac02-2acaf7375f79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.346085 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345868    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.346085 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345889    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51232482-b19e-4fe8-ac02-2acaf7375f79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.346085 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345921    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.346085 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.345958    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-config-volume\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.346384 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:33.346119    2561 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jun 02 15:49:33.346384 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:49:33.346203    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-main-tls podName:51232482-b19e-4fe8-ac02-2acaf7375f79 nodeName:}" failed. No retries permitted until 2026-06-02 15:49:33.846182848 +0000 UTC m=+161.040962332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "51232482-b19e-4fe8-ac02-2acaf7375f79") : secret "alertmanager-main-tls" not found
Jun 02 15:49:33.346861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.346835    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/51232482-b19e-4fe8-ac02-2acaf7375f79-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.347491 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.347465    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51232482-b19e-4fe8-ac02-2acaf7375f79-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.347770 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.347744    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/51232482-b19e-4fe8-ac02-2acaf7375f79-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.348571 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.348527    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/51232482-b19e-4fe8-ac02-2acaf7375f79-config-out\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.349265 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.349241    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/51232482-b19e-4fe8-ac02-2acaf7375f79-tls-assets\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.349406 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.349254    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-config-volume\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.349522 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.349498    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-web-config\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.349709 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.349673    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.349931 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.349912    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.350312 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.350295    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.350663 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.350643    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.358807 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.358785    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctxk\" (UniqueName: \"kubernetes.io/projected/51232482-b19e-4fe8-ac02-2acaf7375f79-kube-api-access-sctxk\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.852371 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.852338    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.854709 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.854689    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/51232482-b19e-4fe8-ac02-2acaf7375f79-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"51232482-b19e-4fe8-ac02-2acaf7375f79\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:33.989109 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.989074    2561 generic.go:358] "Generic (PLEG): container finished" podID="a1460ceb-a365-4964-871a-46ed94304692" containerID="b42e543a12100a4be9af20f21be8eb978a72de4b9bf7acb84ec29c046f655e07" exitCode=0
Jun 02 15:49:33.989268 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:33.989136    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmpcb" event={"ID":"a1460ceb-a365-4964-871a-46ed94304692","Type":"ContainerDied","Data":"b42e543a12100a4be9af20f21be8eb978a72de4b9bf7acb84ec29c046f655e07"}
Jun 02 15:49:34.034749 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:34.034720    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:34.186588 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:34.186558    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:34.189812 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:34.189785    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51232482_b19e_4fe8_ac02_2acaf7375f79.slice/crio-e6fb7fed9b6a6c4101b70bc542afcb49fa60e4ec8b5c7b6cb11650b11ac3b97d WatchSource:0}: Error finding container e6fb7fed9b6a6c4101b70bc542afcb49fa60e4ec8b5c7b6cb11650b11ac3b97d: Status 404 returned error can't find the container with id e6fb7fed9b6a6c4101b70bc542afcb49fa60e4ec8b5c7b6cb11650b11ac3b97d
Jun 02 15:49:34.993439 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:34.993403    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmpcb" event={"ID":"a1460ceb-a365-4964-871a-46ed94304692","Type":"ContainerStarted","Data":"031549c840f44b1959648ad8292a230b61da698ab246e77dbf3ed8226f7141a2"}
Jun 02 15:49:34.993439 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:34.993444    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zmpcb" event={"ID":"a1460ceb-a365-4964-871a-46ed94304692","Type":"ContainerStarted","Data":"ff63e57ca9626bc27f0692522038594bb1afe27e2facc4b54e755ad8c7282cf9"}
Jun 02 15:49:34.995052 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:34.995025    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51232482-b19e-4fe8-ac02-2acaf7375f79","Type":"ContainerStarted","Data":"e6fb7fed9b6a6c4101b70bc542afcb49fa60e4ec8b5c7b6cb11650b11ac3b97d"}
Jun 02 15:49:35.018788 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.018729    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zmpcb" podStartSLOduration=2.273093534 podStartE2EDuration="3.01871317s" podCreationTimestamp="2026-06-02 15:49:32 +0000 UTC" firstStartedPulling="2026-06-02 15:49:32.395407239 +0000 UTC m=+159.590186718" lastFinishedPulling="2026-06-02 15:49:33.141026862 +0000 UTC m=+160.335806354" observedRunningTime="2026-06-02 15:49:35.016721372 +0000 UTC m=+162.211500876" watchObservedRunningTime="2026-06-02 15:49:35.01871317 +0000 UTC m=+162.213492670"
Jun 02 15:49:35.063945 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.063905    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:49:35.066647 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.066622    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"image-registry-659f487f9f-qbcmr\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") " pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:49:35.165673 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.165125    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:49:35.165673 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.165288    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:49:35.168335 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.168299    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5dbb0410-abce-4622-bd7c-dd7ed0faa3b5-metrics-tls\") pod \"dns-default-7tj9z\" (UID: \"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5\") " pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:49:35.168621 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.168603    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc-cert\") pod \"ingress-canary-w2q49\" (UID: \"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc\") " pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:49:35.181370 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.181341    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-25zcr\""
Jun 02 15:49:35.181571 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.181543    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kkg82\""
Jun 02 15:49:35.189139 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.188835    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:49:35.189139 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.188924    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:49:35.385848 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.385816    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-659f487f9f-qbcmr"]
Jun 02 15:49:35.389096 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:35.389065    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5083914_03b4_4dd1_947b_5e55f525b04d.slice/crio-1f2b2febcb8112e87038a92544a92295e6366e995d8daeb10d90192da6136266 WatchSource:0}: Error finding container 1f2b2febcb8112e87038a92544a92295e6366e995d8daeb10d90192da6136266: Status 404 returned error can't find the container with id 1f2b2febcb8112e87038a92544a92295e6366e995d8daeb10d90192da6136266
Jun 02 15:49:35.401876 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.401729    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7tj9z"]
Jun 02 15:49:35.422904 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:35.422879    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dbb0410_abce_4622_bd7c_dd7ed0faa3b5.slice/crio-26d5015377086361b3999f84d94c65e65e771b6a00ec76dea6a718fc0f359cdc WatchSource:0}: Error finding container 26d5015377086361b3999f84d94c65e65e771b6a00ec76dea6a718fc0f359cdc: Status 404 returned error can't find the container with id 26d5015377086361b3999f84d94c65e65e771b6a00ec76dea6a718fc0f359cdc
Jun 02 15:49:35.999177 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.999105    2561 generic.go:358] "Generic (PLEG): container finished" podID="51232482-b19e-4fe8-ac02-2acaf7375f79" containerID="4cf7cb2aaa3dedeabb439692f3ad47971f1f85619ba145af61af99b5d85a7105" exitCode=0
Jun 02 15:49:35.999348 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:35.999193    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51232482-b19e-4fe8-ac02-2acaf7375f79","Type":"ContainerDied","Data":"4cf7cb2aaa3dedeabb439692f3ad47971f1f85619ba145af61af99b5d85a7105"}
Jun 02 15:49:36.000846 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:36.000820    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr" event={"ID":"a5083914-03b4-4dd1-947b-5e55f525b04d","Type":"ContainerStarted","Data":"928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8"}
Jun 02 15:49:36.000948 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:36.000856    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr" event={"ID":"a5083914-03b4-4dd1-947b-5e55f525b04d","Type":"ContainerStarted","Data":"1f2b2febcb8112e87038a92544a92295e6366e995d8daeb10d90192da6136266"}
Jun 02 15:49:36.001022 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:36.000999    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:49:36.002063 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:36.002041    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tj9z" event={"ID":"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5","Type":"ContainerStarted","Data":"26d5015377086361b3999f84d94c65e65e771b6a00ec76dea6a718fc0f359cdc"}
Jun 02 15:49:36.052237 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:36.052181    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr" podStartSLOduration=163.05214856 podStartE2EDuration="2m43.05214856s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:36.051017319 +0000 UTC m=+163.245796845" watchObservedRunningTime="2026-06-02 15:49:36.05214856 +0000 UTC m=+163.246928062"
Jun 02 15:49:37.019477 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:37.019381    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tj9z" event={"ID":"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5","Type":"ContainerStarted","Data":"986ea68e3ebfdc10d6b8da2f4c2f403ee61f36973443d285f4721abf189abdef"}
Jun 02 15:49:37.019477 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:37.019426    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7tj9z" event={"ID":"5dbb0410-abce-4622-bd7c-dd7ed0faa3b5","Type":"ContainerStarted","Data":"c53a607ba7a2210903e85c30fd91d5e4538069843a4a6a8234c7d50a68297d97"}
Jun 02 15:49:38.023640 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.023616    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:49:38.335741 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.335688    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-7tj9z" podStartSLOduration=130.057035653 podStartE2EDuration="2m11.335668039s" podCreationTimestamp="2026-06-02 15:47:27 +0000 UTC" firstStartedPulling="2026-06-02 15:49:35.42468614 +0000 UTC m=+162.619465618" lastFinishedPulling="2026-06-02 15:49:36.703318526 +0000 UTC m=+163.898098004" observedRunningTime="2026-06-02 15:49:37.03979619 +0000 UTC m=+164.234575691" watchObservedRunningTime="2026-06-02 15:49:38.335668039 +0000 UTC m=+165.530447538"
Jun 02 15:49:38.337730 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.337694    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:38.340780 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.340758    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.343851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.343704    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jun 02 15:49:38.343851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.343705    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jun 02 15:49:38.343851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.343739    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jun 02 15:49:38.343851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.343715    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jun 02 15:49:38.343851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.343715    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jun 02 15:49:38.343851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.343806    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-336sarm10t64k\""
Jun 02 15:49:38.343851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.343708    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jun 02 15:49:38.344294 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.344183    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jun 02 15:49:38.344294 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.344260    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jun 02 15:49:38.346368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.344872    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jun 02 15:49:38.346368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.345117    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jun 02 15:49:38.346368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.345318    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jun 02 15:49:38.346368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.345484    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4nbdj\""
Jun 02 15:49:38.346368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.345671    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jun 02 15:49:38.349332 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.348505    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jun 02 15:49:38.358947 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.358924    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:38.473183 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.473082    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8444899ccb-wbzlp"]
Jun 02 15:49:38.499838 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.499812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-web-config\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.499838 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.499841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsfd\" (UniqueName: \"kubernetes.io/projected/39d84e3d-5e9d-498a-8f07-bad27fcc492a-kube-api-access-cwsfd\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.499860    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.499888    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39d84e3d-5e9d-498a-8f07-bad27fcc492a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.499910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.499950    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.499979    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500057 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500104    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500195    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500236    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500269    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-config\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500326    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500679 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39d84e3d-5e9d-498a-8f07-bad27fcc492a-config-out\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500679 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.500679 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.500500    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.600921 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.600886    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.600921 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.600921    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601203 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.600945    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601203 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601085    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601203 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601133    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601210    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601245    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601281    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-config\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601331    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601361    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601610 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601416    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39d84e3d-5e9d-498a-8f07-bad27fcc492a-config-out\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601610 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601447    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601610 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601490    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601610 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601525    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-web-config\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601610 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601550    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsfd\" (UniqueName: \"kubernetes.io/projected/39d84e3d-5e9d-498a-8f07-bad27fcc492a-kube-api-access-cwsfd\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601610 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601577    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601610 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601611    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39d84e3d-5e9d-498a-8f07-bad27fcc492a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601937 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601634    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601937 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601685    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.601937 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.601778    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.604851 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.604821    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.604971 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.604871    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.605293 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.605268    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.606050 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.605769    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.606432 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.606408    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.606675 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.606651    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.607484 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.606993    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.607484 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.607071    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39d84e3d-5e9d-498a-8f07-bad27fcc492a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.607484 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.607353    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39d84e3d-5e9d-498a-8f07-bad27fcc492a-config-out\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.607484 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.607452    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-web-config\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.607684 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.607518    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.607684 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.607527    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39d84e3d-5e9d-498a-8f07-bad27fcc492a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.607950 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.607928    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-config\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.608943 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.608919    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.609043 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.608982    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39d84e3d-5e9d-498a-8f07-bad27fcc492a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.614001 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.613981    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsfd\" (UniqueName: \"kubernetes.io/projected/39d84e3d-5e9d-498a-8f07-bad27fcc492a-kube-api-access-cwsfd\") pod \"prometheus-k8s-0\" (UID: \"39d84e3d-5e9d-498a-8f07-bad27fcc492a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.659592 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.659560    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:38.823816 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:38.823784    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:38.827654 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:38.827614    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d84e3d_5e9d_498a_8f07_bad27fcc492a.slice/crio-0fec2c64da537016d0ab52c17ee51d8e033db384d5358d0e1a0b0b71d19c516d WatchSource:0}: Error finding container 0fec2c64da537016d0ab52c17ee51d8e033db384d5358d0e1a0b0b71d19c516d: Status 404 returned error can't find the container with id 0fec2c64da537016d0ab52c17ee51d8e033db384d5358d0e1a0b0b71d19c516d
Jun 02 15:49:39.027841 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:39.027807    2561 generic.go:358] "Generic (PLEG): container finished" podID="39d84e3d-5e9d-498a-8f07-bad27fcc492a" containerID="a653f3a471fe42dde84388749b1fd26e5ce6dfcd6410963a180b31ad5e0b3ffa" exitCode=0
Jun 02 15:49:39.028235 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:39.027936    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39d84e3d-5e9d-498a-8f07-bad27fcc492a","Type":"ContainerDied","Data":"a653f3a471fe42dde84388749b1fd26e5ce6dfcd6410963a180b31ad5e0b3ffa"}
Jun 02 15:49:39.028235 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:39.027984    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39d84e3d-5e9d-498a-8f07-bad27fcc492a","Type":"ContainerStarted","Data":"0fec2c64da537016d0ab52c17ee51d8e033db384d5358d0e1a0b0b71d19c516d"}
Jun 02 15:49:39.030977 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:39.030943    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51232482-b19e-4fe8-ac02-2acaf7375f79","Type":"ContainerStarted","Data":"24e6647d32b66ef980b7e3a59f78a7350998496fe7eff52fbfc9cd57b8c0389a"}
Jun 02 15:49:39.031063 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:39.030989    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51232482-b19e-4fe8-ac02-2acaf7375f79","Type":"ContainerStarted","Data":"c7507c0a938facd11640ec97b3cc83592d4557f13402ce9ab94cdd1586d6310a"}
Jun 02 15:49:39.031063 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:39.031005    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51232482-b19e-4fe8-ac02-2acaf7375f79","Type":"ContainerStarted","Data":"df5fc9a16fd534ff4a14ce77c1b07352b4bf694ff415267c71369452b578213b"}
Jun 02 15:49:39.031063 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:39.031018    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51232482-b19e-4fe8-ac02-2acaf7375f79","Type":"ContainerStarted","Data":"b0ec4d16d50b93e6a9419cc41c07f12d52bae1930a64992d10510fa4581941e2"}
Jun 02 15:49:39.031063 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:39.031030    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51232482-b19e-4fe8-ac02-2acaf7375f79","Type":"ContainerStarted","Data":"c10a2e05ba2deb30611083580dd59f21c2699cad4c48928f278d1aafa7f73ab9"}
Jun 02 15:49:40.042178 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:40.042128    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"51232482-b19e-4fe8-ac02-2acaf7375f79","Type":"ContainerStarted","Data":"b521d63f73e6cabb5003a6aaf0a0cf7bbc62ee9deb4225101638f9fb6766bfc9"}
Jun 02 15:49:40.072314 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:40.072266    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.232537064 podStartE2EDuration="7.072250784s" podCreationTimestamp="2026-06-02 15:49:33 +0000 UTC" firstStartedPulling="2026-06-02 15:49:34.191724494 +0000 UTC m=+161.386503975" lastFinishedPulling="2026-06-02 15:49:39.031438211 +0000 UTC m=+166.226217695" observedRunningTime="2026-06-02 15:49:40.070955878 +0000 UTC m=+167.265735379" watchObservedRunningTime="2026-06-02 15:49:40.072250784 +0000 UTC m=+167.267030283"
Jun 02 15:49:42.050552 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:42.050480    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39d84e3d-5e9d-498a-8f07-bad27fcc492a","Type":"ContainerStarted","Data":"5c9609ae289cdd66faa7fdb49d4e015b89327c27840f7c486452b11c4a5e4438"}
Jun 02 15:49:42.050552 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:42.050514    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39d84e3d-5e9d-498a-8f07-bad27fcc492a","Type":"ContainerStarted","Data":"bdc6ae69a29e16e114ac02c014ef3e82a01af4aa6f59e2cdc1eedec6c0e673e4"}
Jun 02 15:49:42.428183 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:42.428136    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:49:44.058807 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.058716    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39d84e3d-5e9d-498a-8f07-bad27fcc492a","Type":"ContainerStarted","Data":"dd75f1160eaac88202fda905fcf71423b9495094ed87a094f15f92d9b7da5fae"}
Jun 02 15:49:44.058807 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.058761    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39d84e3d-5e9d-498a-8f07-bad27fcc492a","Type":"ContainerStarted","Data":"0d38bb7da8cbab8fc6feffa9ddab623f50322f74b48d3d33236d8611abd58123"}
Jun 02 15:49:44.058807 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.058773    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39d84e3d-5e9d-498a-8f07-bad27fcc492a","Type":"ContainerStarted","Data":"6fc35ea10e711fdd222c7c385ced4d7a5baa0c3ec17bf8b771b6c495d4d8cdcd"}
Jun 02 15:49:44.058807 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.058787    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39d84e3d-5e9d-498a-8f07-bad27fcc492a","Type":"ContainerStarted","Data":"e56922a8e0474a6cdfb154e456cd043481bd838bd8d2c52274cc75456e966357"}
Jun 02 15:49:44.098554 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.098499    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.573484831 podStartE2EDuration="6.098481314s" podCreationTimestamp="2026-06-02 15:49:38 +0000 UTC" firstStartedPulling="2026-06-02 15:49:39.029254164 +0000 UTC m=+166.224033642" lastFinishedPulling="2026-06-02 15:49:43.554250641 +0000 UTC m=+170.749030125" observedRunningTime="2026-06-02 15:49:44.095801287 +0000 UTC m=+171.290580787" watchObservedRunningTime="2026-06-02 15:49:44.098481314 +0000 UTC m=+171.293260815"
Jun 02 15:49:44.428995 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.428962    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:49:44.432144 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.432122    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gg4df\""
Jun 02 15:49:44.440416 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.440391    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w2q49"
Jun 02 15:49:44.584735 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:44.584679    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w2q49"]
Jun 02 15:49:44.589042 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:49:44.589010    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ba04ba_dd8d_4f15_9629_f84b2f6e81dc.slice/crio-e921f9a168c379e060d6600d1f3f9cd3a066e1f05b8836316d8654f1270787d8 WatchSource:0}: Error finding container e921f9a168c379e060d6600d1f3f9cd3a066e1f05b8836316d8654f1270787d8: Status 404 returned error can't find the container with id e921f9a168c379e060d6600d1f3f9cd3a066e1f05b8836316d8654f1270787d8
Jun 02 15:49:45.063511 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:45.063468    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w2q49" event={"ID":"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc","Type":"ContainerStarted","Data":"e921f9a168c379e060d6600d1f3f9cd3a066e1f05b8836316d8654f1270787d8"}
Jun 02 15:49:47.072451 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:47.072418    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w2q49" event={"ID":"a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc","Type":"ContainerStarted","Data":"58937e6af8c546044b6877ba6660bb08c257050ea916114fc047a6f1e526fd59"}
Jun 02 15:49:47.094588 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:47.094543    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w2q49" podStartSLOduration=138.674328443 podStartE2EDuration="2m20.094529656s" podCreationTimestamp="2026-06-02 15:47:27 +0000 UTC" firstStartedPulling="2026-06-02 15:49:44.591059563 +0000 UTC m=+171.785839044" lastFinishedPulling="2026-06-02 15:49:46.011260773 +0000 UTC m=+173.206040257" observedRunningTime="2026-06-02 15:49:47.092273789 +0000 UTC m=+174.287053288" watchObservedRunningTime="2026-06-02 15:49:47.094529656 +0000 UTC m=+174.289309156"
Jun 02 15:49:48.032765 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:48.032733    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7tj9z"
Jun 02 15:49:48.660102 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:48.660062    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:50.569531 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:50.569496    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-659f487f9f-qbcmr"]
Jun 02 15:49:50.573459 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:50.573425    2561 patch_prober.go:28] interesting pod/image-registry-659f487f9f-qbcmr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 02 15:49:50.573608 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:49:50.573491    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr" podUID="a5083914-03b4-4dd1-947b-5e55f525b04d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 15:50:00.573315 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:00.573285    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:50:03.495749 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.495687    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-8444899ccb-wbzlp" podUID="850916db-3f77-4547-9a28-8a034d41704f" containerName="console" containerID="cri-o://a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1" gracePeriod=15
Jun 02 15:50:03.733513 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.733491    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8444899ccb-wbzlp_850916db-3f77-4547-9a28-8a034d41704f/console/0.log"
Jun 02 15:50:03.733636 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.733566    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:50:03.831924 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.831855    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-oauth-config\") pod \"850916db-3f77-4547-9a28-8a034d41704f\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") "
Jun 02 15:50:03.831924 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.831895    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-console-config\") pod \"850916db-3f77-4547-9a28-8a034d41704f\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") "
Jun 02 15:50:03.832111 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.831943    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-service-ca\") pod \"850916db-3f77-4547-9a28-8a034d41704f\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") "
Jun 02 15:50:03.832111 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.832004    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-oauth-serving-cert\") pod \"850916db-3f77-4547-9a28-8a034d41704f\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") "
Jun 02 15:50:03.832111 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.832022    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-serving-cert\") pod \"850916db-3f77-4547-9a28-8a034d41704f\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") "
Jun 02 15:50:03.832111 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.832040    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmk4d\" (UniqueName: \"kubernetes.io/projected/850916db-3f77-4547-9a28-8a034d41704f-kube-api-access-wmk4d\") pod \"850916db-3f77-4547-9a28-8a034d41704f\" (UID: \"850916db-3f77-4547-9a28-8a034d41704f\") "
Jun 02 15:50:03.832380 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.832351    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-console-config" (OuterVolumeSpecName: "console-config") pod "850916db-3f77-4547-9a28-8a034d41704f" (UID: "850916db-3f77-4547-9a28-8a034d41704f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:03.832479 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.832401    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "850916db-3f77-4547-9a28-8a034d41704f" (UID: "850916db-3f77-4547-9a28-8a034d41704f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:03.832479 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.832400    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-service-ca" (OuterVolumeSpecName: "service-ca") pod "850916db-3f77-4547-9a28-8a034d41704f" (UID: "850916db-3f77-4547-9a28-8a034d41704f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:03.834148 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.834122    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "850916db-3f77-4547-9a28-8a034d41704f" (UID: "850916db-3f77-4547-9a28-8a034d41704f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:50:03.834264 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.834172    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "850916db-3f77-4547-9a28-8a034d41704f" (UID: "850916db-3f77-4547-9a28-8a034d41704f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:50:03.834309 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.834274    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850916db-3f77-4547-9a28-8a034d41704f-kube-api-access-wmk4d" (OuterVolumeSpecName: "kube-api-access-wmk4d") pod "850916db-3f77-4547-9a28-8a034d41704f" (UID: "850916db-3f77-4547-9a28-8a034d41704f"). InnerVolumeSpecName "kube-api-access-wmk4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:50:03.932931 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.932900    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-service-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:03.932931 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.932929    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-oauth-serving-cert\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:03.932931 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.932939    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-serving-cert\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:03.933202 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.932948    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmk4d\" (UniqueName: \"kubernetes.io/projected/850916db-3f77-4547-9a28-8a034d41704f-kube-api-access-wmk4d\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:03.933202 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.932961    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/850916db-3f77-4547-9a28-8a034d41704f-console-oauth-config\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:03.933202 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:03.932969    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/850916db-3f77-4547-9a28-8a034d41704f-console-config\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:04.121756 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.121690    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8444899ccb-wbzlp_850916db-3f77-4547-9a28-8a034d41704f/console/0.log"
Jun 02 15:50:04.121756 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.121725    2561 generic.go:358] "Generic (PLEG): container finished" podID="850916db-3f77-4547-9a28-8a034d41704f" containerID="a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1" exitCode=2
Jun 02 15:50:04.121914 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.121760    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8444899ccb-wbzlp" event={"ID":"850916db-3f77-4547-9a28-8a034d41704f","Type":"ContainerDied","Data":"a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1"}
Jun 02 15:50:04.121914 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.121782    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8444899ccb-wbzlp"
Jun 02 15:50:04.121914 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.121797    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8444899ccb-wbzlp" event={"ID":"850916db-3f77-4547-9a28-8a034d41704f","Type":"ContainerDied","Data":"013d505d262dddd485a8e05c71c1939d4b8b8c71a2a86b2bed7764d44c0577cc"}
Jun 02 15:50:04.121914 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.121816    2561 scope.go:117] "RemoveContainer" containerID="a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1"
Jun 02 15:50:04.129930 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.129915    2561 scope.go:117] "RemoveContainer" containerID="a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1"
Jun 02 15:50:04.130187 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:50:04.130148    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1\": container with ID starting with a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1 not found: ID does not exist" containerID="a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1"
Jun 02 15:50:04.130261 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.130193    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1"} err="failed to get container status \"a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1\": rpc error: code = NotFound desc = could not find container \"a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1\": container with ID starting with a1358113792f429d37e0710d8e725bc421bf0da89712180e9c792c0d098939b1 not found: ID does not exist"
Jun 02 15:50:04.143706 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.143683    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8444899ccb-wbzlp"]
Jun 02 15:50:04.148853 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:04.148829    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8444899ccb-wbzlp"]
Jun 02 15:50:05.432212 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:05.432176    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850916db-3f77-4547-9a28-8a034d41704f" path="/var/lib/kubelet/pods/850916db-3f77-4547-9a28-8a034d41704f/volumes"
Jun 02 15:50:09.139843 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:09.139812    2561 generic.go:358] "Generic (PLEG): container finished" podID="64f59933-6166-4e8c-b115-ac029db80096" containerID="c37a3584c15d190a5539599e9a2eedd0fdce97c1886641a1af5bb1a3ec933bb8" exitCode=0
Jun 02 15:50:09.140134 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:09.139884    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk" event={"ID":"64f59933-6166-4e8c-b115-ac029db80096","Type":"ContainerDied","Data":"c37a3584c15d190a5539599e9a2eedd0fdce97c1886641a1af5bb1a3ec933bb8"}
Jun 02 15:50:09.140214 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:09.140199    2561 scope.go:117] "RemoveContainer" containerID="c37a3584c15d190a5539599e9a2eedd0fdce97c1886641a1af5bb1a3ec933bb8"
Jun 02 15:50:10.144144 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:10.144112    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-96ddc787c-pn6mk" event={"ID":"64f59933-6166-4e8c-b115-ac029db80096","Type":"ContainerStarted","Data":"2b712c398405748fe6552dbde8515e6f4fec7a857d24d06fb6aa6fbcf94fdfc2"}
Jun 02 15:50:15.158925 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.158890    2561 generic.go:358] "Generic (PLEG): container finished" podID="0a46d01f-06de-4eaf-9762-c1c02fd68d11" containerID="4cc09a7382a75ed535ed6decea611e76ff51134614eeca5ef10579d78d03469a" exitCode=0
Jun 02 15:50:15.159311 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.158968    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp" event={"ID":"0a46d01f-06de-4eaf-9762-c1c02fd68d11","Type":"ContainerDied","Data":"4cc09a7382a75ed535ed6decea611e76ff51134614eeca5ef10579d78d03469a"}
Jun 02 15:50:15.159354 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.159325    2561 scope.go:117] "RemoveContainer" containerID="4cc09a7382a75ed535ed6decea611e76ff51134614eeca5ef10579d78d03469a"
Jun 02 15:50:15.589546 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.589437    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr" podUID="a5083914-03b4-4dd1-947b-5e55f525b04d" containerName="registry" containerID="cri-o://928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8" gracePeriod=30
Jun 02 15:50:15.847498 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.847441    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:50:15.938328 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938295    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-certificates\") pod \"a5083914-03b4-4dd1-947b-5e55f525b04d\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") "
Jun 02 15:50:15.938509 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938366    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-installation-pull-secrets\") pod \"a5083914-03b4-4dd1-947b-5e55f525b04d\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") "
Jun 02 15:50:15.938509 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938414    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-trusted-ca\") pod \"a5083914-03b4-4dd1-947b-5e55f525b04d\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") "
Jun 02 15:50:15.938509 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938439    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sqq6\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-kube-api-access-5sqq6\") pod \"a5083914-03b4-4dd1-947b-5e55f525b04d\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") "
Jun 02 15:50:15.938509 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938466    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") pod \"a5083914-03b4-4dd1-947b-5e55f525b04d\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") "
Jun 02 15:50:15.938714 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938512    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-bound-sa-token\") pod \"a5083914-03b4-4dd1-947b-5e55f525b04d\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") "
Jun 02 15:50:15.938714 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938543    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5083914-03b4-4dd1-947b-5e55f525b04d-ca-trust-extracted\") pod \"a5083914-03b4-4dd1-947b-5e55f525b04d\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") "
Jun 02 15:50:15.938714 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938597    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-image-registry-private-configuration\") pod \"a5083914-03b4-4dd1-947b-5e55f525b04d\" (UID: \"a5083914-03b4-4dd1-947b-5e55f525b04d\") "
Jun 02 15:50:15.938854 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938795    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a5083914-03b4-4dd1-947b-5e55f525b04d" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:15.938913 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938874    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-certificates\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:15.939280 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.938876    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a5083914-03b4-4dd1-947b-5e55f525b04d" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:50:15.946735 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.946667    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-kube-api-access-5sqq6" (OuterVolumeSpecName: "kube-api-access-5sqq6") pod "a5083914-03b4-4dd1-947b-5e55f525b04d" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d"). InnerVolumeSpecName "kube-api-access-5sqq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:50:15.947967 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.947311    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a5083914-03b4-4dd1-947b-5e55f525b04d" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:50:15.947967 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.947539    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a5083914-03b4-4dd1-947b-5e55f525b04d" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:50:15.948485 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.948188    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a5083914-03b4-4dd1-947b-5e55f525b04d" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:50:15.948789 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.948766    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a5083914-03b4-4dd1-947b-5e55f525b04d" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:50:15.951530 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:15.951502    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5083914-03b4-4dd1-947b-5e55f525b04d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a5083914-03b4-4dd1-947b-5e55f525b04d" (UID: "a5083914-03b4-4dd1-947b-5e55f525b04d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:50:16.039864 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.039830    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-installation-pull-secrets\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:16.039864 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.039858    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a5083914-03b4-4dd1-947b-5e55f525b04d-trusted-ca\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:16.039864 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.039868    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sqq6\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-kube-api-access-5sqq6\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:16.040099 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.039878    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-registry-tls\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:16.040099 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.039887    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a5083914-03b4-4dd1-947b-5e55f525b04d-bound-sa-token\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:16.040099 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.039894    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a5083914-03b4-4dd1-947b-5e55f525b04d-ca-trust-extracted\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:16.040099 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.039904    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a5083914-03b4-4dd1-947b-5e55f525b04d-image-registry-private-configuration\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 15:50:16.162839 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.162795    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-69877444c-s8fhp" event={"ID":"0a46d01f-06de-4eaf-9762-c1c02fd68d11","Type":"ContainerStarted","Data":"12c93da16ffe88ad53a80840f19fcb2b72ce2e09f0a88ae3c0a0db3d523325e8"}
Jun 02 15:50:16.163903 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.163877    2561 generic.go:358] "Generic (PLEG): container finished" podID="a5083914-03b4-4dd1-947b-5e55f525b04d" containerID="928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8" exitCode=0
Jun 02 15:50:16.164025 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.163988    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr" event={"ID":"a5083914-03b4-4dd1-947b-5e55f525b04d","Type":"ContainerDied","Data":"928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8"}
Jun 02 15:50:16.164025 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.164008    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr" event={"ID":"a5083914-03b4-4dd1-947b-5e55f525b04d","Type":"ContainerDied","Data":"1f2b2febcb8112e87038a92544a92295e6366e995d8daeb10d90192da6136266"}
Jun 02 15:50:16.164025 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.164022    2561 scope.go:117] "RemoveContainer" containerID="928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8"
Jun 02 15:50:16.164205 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.164021    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-659f487f9f-qbcmr"
Jun 02 15:50:16.172322 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.172305    2561 scope.go:117] "RemoveContainer" containerID="928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8"
Jun 02 15:50:16.172608 ip-10-0-135-231 kubenswrapper[2561]: E0602 15:50:16.172587    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8\": container with ID starting with 928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8 not found: ID does not exist" containerID="928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8"
Jun 02 15:50:16.172665 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.172615    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8"} err="failed to get container status \"928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8\": rpc error: code = NotFound desc = could not find container \"928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8\": container with ID starting with 928005ed18618c9ec63e536862b7cd6285d12513d6d75bca6f3ae2cb07eff5f8 not found: ID does not exist"
Jun 02 15:50:16.199566 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.199535    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-659f487f9f-qbcmr"]
Jun 02 15:50:16.210441 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:16.210417    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-659f487f9f-qbcmr"]
Jun 02 15:50:17.432368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:17.432334    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5083914-03b4-4dd1-947b-5e55f525b04d" path="/var/lib/kubelet/pods/a5083914-03b4-4dd1-947b-5e55f525b04d/volumes"
Jun 02 15:50:20.175679 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:20.175649    2561 generic.go:358] "Generic (PLEG): container finished" podID="8ca99b2d-46f2-454b-89eb-df55517239d0" containerID="5f012ee8d253147d23b3ba45fa876403ea6bb08bfdc9943187382d3aab891c9c" exitCode=0
Jun 02 15:50:20.176036 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:20.175704    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-788bf58df5-lv6l8" event={"ID":"8ca99b2d-46f2-454b-89eb-df55517239d0","Type":"ContainerDied","Data":"5f012ee8d253147d23b3ba45fa876403ea6bb08bfdc9943187382d3aab891c9c"}
Jun 02 15:50:20.176036 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:20.176002    2561 scope.go:117] "RemoveContainer" containerID="5f012ee8d253147d23b3ba45fa876403ea6bb08bfdc9943187382d3aab891c9c"
Jun 02 15:50:21.180090 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:21.180058    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-788bf58df5-lv6l8" event={"ID":"8ca99b2d-46f2-454b-89eb-df55517239d0","Type":"ContainerStarted","Data":"a40fe33253a0685eea6d5ec2fc0332d577be12e9651bcd5eab1cd91bbb8f0470"}
Jun 02 15:50:38.660561 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:38.660509    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:50:38.675999 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:38.675974    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:50:39.247199 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:39.247151    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:50:56.651368 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.651331    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"]
Jun 02 15:50:56.651850 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.651724    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5083914-03b4-4dd1-947b-5e55f525b04d" containerName="registry"
Jun 02 15:50:56.651850 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.651742    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5083914-03b4-4dd1-947b-5e55f525b04d" containerName="registry"
Jun 02 15:50:56.651850 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.651752    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850916db-3f77-4547-9a28-8a034d41704f" containerName="console"
Jun 02 15:50:56.651850 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.651761    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="850916db-3f77-4547-9a28-8a034d41704f" containerName="console"
Jun 02 15:50:56.652043 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.651863    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="a5083914-03b4-4dd1-947b-5e55f525b04d" containerName="registry"
Jun 02 15:50:56.652043 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.651878    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="850916db-3f77-4547-9a28-8a034d41704f" containerName="console"
Jun 02 15:50:56.655256 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.655236    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.658032 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.658010    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jun 02 15:50:56.658156 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.658059    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jun 02 15:50:56.658156 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.658034    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jun 02 15:50:56.658156 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.658105    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jun 02 15:50:56.658403 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.658356    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s5xxc\""
Jun 02 15:50:56.658403 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.658361    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jun 02 15:50:56.663838 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.663812    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jun 02 15:50:56.666123 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.666103    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"]
Jun 02 15:50:56.788303 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.788268    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.788450 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.788349    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-secret-telemeter-client\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.788450 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.788378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.788450 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.788410    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-federate-client-tls\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.788450 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.788435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-serving-certs-ca-bundle\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.788580 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.788490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-metrics-client-ca\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.788580 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.788522    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6vk9\" (UniqueName: \"kubernetes.io/projected/25210a6c-f667-42dd-8b97-1a4fda7ab373-kube-api-access-z6vk9\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.788641 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.788581    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-telemeter-client-tls\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.889112 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.889077    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.889290 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.889122    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-secret-telemeter-client\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.889290 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.889150    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.889290 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.889196    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-federate-client-tls\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.889290 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.889217    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-serving-certs-ca-bundle\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.889290 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.889245    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-metrics-client-ca\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.889290 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.889273    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z6vk9\" (UniqueName: \"kubernetes.io/projected/25210a6c-f667-42dd-8b97-1a4fda7ab373-kube-api-access-z6vk9\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.889563 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.889432    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-telemeter-client-tls\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.890128 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.890099    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-metrics-client-ca\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.890274 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.890232    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.890599 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.890573    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25210a6c-f667-42dd-8b97-1a4fda7ab373-serving-certs-ca-bundle\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.891762 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.891742    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-secret-telemeter-client\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.891936 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.891916    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-telemeter-client-tls\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.892043 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.892027    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.892080 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.892054    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/25210a6c-f667-42dd-8b97-1a4fda7ab373-federate-client-tls\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.898263 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.898245    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6vk9\" (UniqueName: \"kubernetes.io/projected/25210a6c-f667-42dd-8b97-1a4fda7ab373-kube-api-access-z6vk9\") pod \"telemeter-client-54b7b45c56-cgqrn\" (UID: \"25210a6c-f667-42dd-8b97-1a4fda7ab373\") " pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:56.964901 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:56.964837    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"
Jun 02 15:50:57.099650 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:57.098016    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54b7b45c56-cgqrn"]
Jun 02 15:50:57.101963 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:50:57.101935    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25210a6c_f667_42dd_8b97_1a4fda7ab373.slice/crio-d006cac182875b9a23a3805808387fab35f0e6a966d73099005017c30ea9fbdb WatchSource:0}: Error finding container d006cac182875b9a23a3805808387fab35f0e6a966d73099005017c30ea9fbdb: Status 404 returned error can't find the container with id d006cac182875b9a23a3805808387fab35f0e6a966d73099005017c30ea9fbdb
Jun 02 15:50:57.284009 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:57.283934    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn" event={"ID":"25210a6c-f667-42dd-8b97-1a4fda7ab373","Type":"ContainerStarted","Data":"d006cac182875b9a23a3805808387fab35f0e6a966d73099005017c30ea9fbdb"}
Jun 02 15:50:59.292051 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:59.291973    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn" event={"ID":"25210a6c-f667-42dd-8b97-1a4fda7ab373","Type":"ContainerStarted","Data":"2a5c0afa1fb889e2dcb6dd330140679534a3c7355141ff6e4d369857fe101254"}
Jun 02 15:50:59.292051 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:59.292009    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn" event={"ID":"25210a6c-f667-42dd-8b97-1a4fda7ab373","Type":"ContainerStarted","Data":"8e22e011013a69a44e02ca259491f36f733a7e38d54040f340066b1d206fd39a"}
Jun 02 15:50:59.292051 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:59.292019    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn" event={"ID":"25210a6c-f667-42dd-8b97-1a4fda7ab373","Type":"ContainerStarted","Data":"6fdecd8e6946a084aadfba55cd57a0378c83fc11722f09b51df2ae70635c7010"}
Jun 02 15:50:59.320865 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:50:59.320817    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54b7b45c56-cgqrn" podStartSLOduration=1.449951478 podStartE2EDuration="3.320804752s" podCreationTimestamp="2026-06-02 15:50:56 +0000 UTC" firstStartedPulling="2026-06-02 15:50:57.103738115 +0000 UTC m=+244.298517594" lastFinishedPulling="2026-06-02 15:50:58.974591389 +0000 UTC m=+246.169370868" observedRunningTime="2026-06-02 15:50:59.319522979 +0000 UTC m=+246.514302523" watchObservedRunningTime="2026-06-02 15:50:59.320804752 +0000 UTC m=+246.515584252"
Jun 02 15:51:05.261553 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:05.261518    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:51:05.263740 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:05.263722    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d86be44-4451-4903-bf56-5179cec7698d-metrics-certs\") pod \"network-metrics-daemon-48g4z\" (UID: \"6d86be44-4451-4903-bf56-5179cec7698d\") " pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:51:05.532266 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:05.532177    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j9fj4\""
Jun 02 15:51:05.539526 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:05.539505    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48g4z"
Jun 02 15:51:05.663027 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:05.662994    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48g4z"]
Jun 02 15:51:05.666061 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:51:05.666041    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d86be44_4451_4903_bf56_5179cec7698d.slice/crio-aab2d98b31edeed10f675ea854c0df8f716a0367f346605dca4f122bdd5753f5 WatchSource:0}: Error finding container aab2d98b31edeed10f675ea854c0df8f716a0367f346605dca4f122bdd5753f5: Status 404 returned error can't find the container with id aab2d98b31edeed10f675ea854c0df8f716a0367f346605dca4f122bdd5753f5
Jun 02 15:51:06.313589 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:06.313544    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48g4z" event={"ID":"6d86be44-4451-4903-bf56-5179cec7698d","Type":"ContainerStarted","Data":"aab2d98b31edeed10f675ea854c0df8f716a0367f346605dca4f122bdd5753f5"}
Jun 02 15:51:07.319471 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:07.319429    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48g4z" event={"ID":"6d86be44-4451-4903-bf56-5179cec7698d","Type":"ContainerStarted","Data":"d73a0dcf9074ea093cb5fa920317116df29d8a44ccf5a1e889287e6a2874fca7"}
Jun 02 15:51:07.319471 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:07.319469    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48g4z" event={"ID":"6d86be44-4451-4903-bf56-5179cec7698d","Type":"ContainerStarted","Data":"835aeb24433911a9bdd6895db62b794cb95c0d7f1e39645959aeddd5f340d79f"}
Jun 02 15:51:07.341265 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:07.341223    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-48g4z" podStartSLOduration=253.325721134 podStartE2EDuration="4m14.341209653s" podCreationTimestamp="2026-06-02 15:46:53 +0000 UTC" firstStartedPulling="2026-06-02 15:51:05.667860542 +0000 UTC m=+252.862640020" lastFinishedPulling="2026-06-02 15:51:06.683349057 +0000 UTC m=+253.878128539" observedRunningTime="2026-06-02 15:51:07.33901568 +0000 UTC m=+254.533795196" watchObservedRunningTime="2026-06-02 15:51:07.341209653 +0000 UTC m=+254.535989153"
Jun 02 15:51:53.318065 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:53.318034    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 15:51:53.318065 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:53.318045    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 15:51:53.324797 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:53.324771    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 15:51:53.324951 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:53.324911    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 15:51:53.328286 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:51:53.328265    2561 kubelet.go:1628] "Image garbage collection succeeded"
Jun 02 15:54:20.429225 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.429132    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-7kzhd"]
Jun 02 15:54:20.431381 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.431365    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:20.436923 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.436901    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gpmhp\""
Jun 02 15:54:20.437150 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.437130    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jun 02 15:54:20.438344 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.438326    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jun 02 15:54:20.446332 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.446307    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-7kzhd"]
Jun 02 15:54:20.503637 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.503611    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d928dc88-a053-4106-8d42-94781b1acc47-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-7kzhd\" (UID: \"d928dc88-a053-4106-8d42-94781b1acc47\") " pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:20.503778 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.503667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c76f\" (UniqueName: \"kubernetes.io/projected/d928dc88-a053-4106-8d42-94781b1acc47-kube-api-access-7c76f\") pod \"cert-manager-webhook-587ccfb98-7kzhd\" (UID: \"d928dc88-a053-4106-8d42-94781b1acc47\") " pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:20.604924 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.604869    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d928dc88-a053-4106-8d42-94781b1acc47-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-7kzhd\" (UID: \"d928dc88-a053-4106-8d42-94781b1acc47\") " pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:20.605094 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.604940    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7c76f\" (UniqueName: \"kubernetes.io/projected/d928dc88-a053-4106-8d42-94781b1acc47-kube-api-access-7c76f\") pod \"cert-manager-webhook-587ccfb98-7kzhd\" (UID: \"d928dc88-a053-4106-8d42-94781b1acc47\") " pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:20.614979 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.614948    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d928dc88-a053-4106-8d42-94781b1acc47-bound-sa-token\") pod \"cert-manager-webhook-587ccfb98-7kzhd\" (UID: \"d928dc88-a053-4106-8d42-94781b1acc47\") " pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:20.615407 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.615383    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c76f\" (UniqueName: \"kubernetes.io/projected/d928dc88-a053-4106-8d42-94781b1acc47-kube-api-access-7c76f\") pod \"cert-manager-webhook-587ccfb98-7kzhd\" (UID: \"d928dc88-a053-4106-8d42-94781b1acc47\") " pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:20.760817 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.760728    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:20.889145 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.889114    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-587ccfb98-7kzhd"]
Jun 02 15:54:20.893470 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:54:20.893443    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd928dc88_a053_4106_8d42_94781b1acc47.slice/crio-44dbe49acaf808f55ffe0181fd9bc0125a126b776eeb58381461c77f17d35738 WatchSource:0}: Error finding container 44dbe49acaf808f55ffe0181fd9bc0125a126b776eeb58381461c77f17d35738: Status 404 returned error can't find the container with id 44dbe49acaf808f55ffe0181fd9bc0125a126b776eeb58381461c77f17d35738
Jun 02 15:54:20.895174 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:20.895145    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:54:21.882400 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:21.882334    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd" event={"ID":"d928dc88-a053-4106-8d42-94781b1acc47","Type":"ContainerStarted","Data":"44dbe49acaf808f55ffe0181fd9bc0125a126b776eeb58381461c77f17d35738"}
Jun 02 15:54:23.889861 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:23.889825    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd" event={"ID":"d928dc88-a053-4106-8d42-94781b1acc47","Type":"ContainerStarted","Data":"51d4a48f27ca09d68963bda06b068d29984388a0883c4d3ab36243c5f7bad19d"}
Jun 02 15:54:23.890291 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:23.889887    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:23.909185 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:23.909100    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd" podStartSLOduration=1.124648559 podStartE2EDuration="3.909085263s" podCreationTimestamp="2026-06-02 15:54:20 +0000 UTC" firstStartedPulling="2026-06-02 15:54:20.895360622 +0000 UTC m=+448.090140117" lastFinishedPulling="2026-06-02 15:54:23.679797342 +0000 UTC m=+450.874576821" observedRunningTime="2026-06-02 15:54:23.907770398 +0000 UTC m=+451.102549897" watchObservedRunningTime="2026-06-02 15:54:23.909085263 +0000 UTC m=+451.103864763"
Jun 02 15:54:29.894921 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:29.894884    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-587ccfb98-7kzhd"
Jun 02 15:54:33.478981 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.478940    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-79c8d999ff-b6d95"]
Jun 02 15:54:33.482487 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.482466    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-b6d95"
Jun 02 15:54:33.487837 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.487814    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ns6nc\""
Jun 02 15:54:33.508051 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.508026    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-b6d95"]
Jun 02 15:54:33.621705 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.621668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf5kx\" (UniqueName: \"kubernetes.io/projected/44e14896-78e7-4e3b-b2cd-16aee3f4843a-kube-api-access-tf5kx\") pod \"cert-manager-79c8d999ff-b6d95\" (UID: \"44e14896-78e7-4e3b-b2cd-16aee3f4843a\") " pod="cert-manager/cert-manager-79c8d999ff-b6d95"
Jun 02 15:54:33.621868 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.621768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44e14896-78e7-4e3b-b2cd-16aee3f4843a-bound-sa-token\") pod \"cert-manager-79c8d999ff-b6d95\" (UID: \"44e14896-78e7-4e3b-b2cd-16aee3f4843a\") " pod="cert-manager/cert-manager-79c8d999ff-b6d95"
Jun 02 15:54:33.723103 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.723058    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tf5kx\" (UniqueName: \"kubernetes.io/projected/44e14896-78e7-4e3b-b2cd-16aee3f4843a-kube-api-access-tf5kx\") pod \"cert-manager-79c8d999ff-b6d95\" (UID: \"44e14896-78e7-4e3b-b2cd-16aee3f4843a\") " pod="cert-manager/cert-manager-79c8d999ff-b6d95"
Jun 02 15:54:33.723283 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.723183    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44e14896-78e7-4e3b-b2cd-16aee3f4843a-bound-sa-token\") pod \"cert-manager-79c8d999ff-b6d95\" (UID: \"44e14896-78e7-4e3b-b2cd-16aee3f4843a\") " pod="cert-manager/cert-manager-79c8d999ff-b6d95"
Jun 02 15:54:33.733775 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.733700    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44e14896-78e7-4e3b-b2cd-16aee3f4843a-bound-sa-token\") pod \"cert-manager-79c8d999ff-b6d95\" (UID: \"44e14896-78e7-4e3b-b2cd-16aee3f4843a\") " pod="cert-manager/cert-manager-79c8d999ff-b6d95"
Jun 02 15:54:33.733906 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.733837    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf5kx\" (UniqueName: \"kubernetes.io/projected/44e14896-78e7-4e3b-b2cd-16aee3f4843a-kube-api-access-tf5kx\") pod \"cert-manager-79c8d999ff-b6d95\" (UID: \"44e14896-78e7-4e3b-b2cd-16aee3f4843a\") " pod="cert-manager/cert-manager-79c8d999ff-b6d95"
Jun 02 15:54:33.791134 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.791086    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-b6d95"
Jun 02 15:54:33.923838 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:33.923806    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-b6d95"]
Jun 02 15:54:33.926819 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:54:33.926788    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44e14896_78e7_4e3b_b2cd_16aee3f4843a.slice/crio-8340bd504c905c518ebf14cc53efd90839dcf26ffde75bad581fa151a4bca043 WatchSource:0}: Error finding container 8340bd504c905c518ebf14cc53efd90839dcf26ffde75bad581fa151a4bca043: Status 404 returned error can't find the container with id 8340bd504c905c518ebf14cc53efd90839dcf26ffde75bad581fa151a4bca043
Jun 02 15:54:34.922987 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:34.922951    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-b6d95" event={"ID":"44e14896-78e7-4e3b-b2cd-16aee3f4843a","Type":"ContainerStarted","Data":"9cad7e55da0b57fe415d554e02934e3e1b3cfefcc7d61992e6f746aa8f029d37"}
Jun 02 15:54:34.922987 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:34.922989    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-b6d95" event={"ID":"44e14896-78e7-4e3b-b2cd-16aee3f4843a","Type":"ContainerStarted","Data":"8340bd504c905c518ebf14cc53efd90839dcf26ffde75bad581fa151a4bca043"}
Jun 02 15:54:34.944041 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:54:34.943987    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-79c8d999ff-b6d95" podStartSLOduration=1.943970598 podStartE2EDuration="1.943970598s" podCreationTimestamp="2026-06-02 15:54:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:54:34.942628642 +0000 UTC m=+462.137408143" watchObservedRunningTime="2026-06-02 15:54:34.943970598 +0000 UTC m=+462.138750098"
Jun 02 15:55:05.911088 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.911052    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"]
Jun 02 15:55:05.915966 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.915944    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:05.920642 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.920612    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:55:05.920642 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.920629    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jun 02 15:55:05.921962 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.921941    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-pzsd6\""
Jun 02 15:55:05.922076 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.921988    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jun 02 15:55:05.922141 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.922125    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jun 02 15:55:05.922248 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.922231    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jun 02 15:55:05.933325 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:05.933294    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"]
Jun 02 15:55:06.002274 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.002235    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpm7l\" (UniqueName: \"kubernetes.io/projected/9fe3052e-1694-446e-9f09-607c2f07f746-kube-api-access-kpm7l\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.002451 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.002291    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fe3052e-1694-446e-9f09-607c2f07f746-cert\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.002451 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.002311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fe3052e-1694-446e-9f09-607c2f07f746-metrics-cert\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.002451 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.002341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fe3052e-1694-446e-9f09-607c2f07f746-manager-config\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.102957 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.102916    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fe3052e-1694-446e-9f09-607c2f07f746-manager-config\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.103144 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.102992    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm7l\" (UniqueName: \"kubernetes.io/projected/9fe3052e-1694-446e-9f09-607c2f07f746-kube-api-access-kpm7l\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.103144 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.103040    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fe3052e-1694-446e-9f09-607c2f07f746-cert\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.103144 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.103064    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fe3052e-1694-446e-9f09-607c2f07f746-metrics-cert\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.103631 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.103598    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fe3052e-1694-446e-9f09-607c2f07f746-manager-config\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.105536 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.105518    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fe3052e-1694-446e-9f09-607c2f07f746-cert\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.105630 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.105577    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fe3052e-1694-446e-9f09-607c2f07f746-metrics-cert\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.116980 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.116948    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm7l\" (UniqueName: \"kubernetes.io/projected/9fe3052e-1694-446e-9f09-607c2f07f746-kube-api-access-kpm7l\") pod \"lws-controller-manager-74c68556d6-bshb6\" (UID: \"9fe3052e-1694-446e-9f09-607c2f07f746\") " pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.225267 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.225177    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:06.362069 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:06.362037    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"]
Jun 02 15:55:06.365650 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:55:06.365623    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe3052e_1694_446e_9f09_607c2f07f746.slice/crio-eb924b0de4f0172769621d6d514dd8de26d619929a44d7d7170c79f4d91be1e6 WatchSource:0}: Error finding container eb924b0de4f0172769621d6d514dd8de26d619929a44d7d7170c79f4d91be1e6: Status 404 returned error can't find the container with id eb924b0de4f0172769621d6d514dd8de26d619929a44d7d7170c79f4d91be1e6
Jun 02 15:55:07.018731 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:07.018700    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6" event={"ID":"9fe3052e-1694-446e-9f09-607c2f07f746","Type":"ContainerStarted","Data":"eb924b0de4f0172769621d6d514dd8de26d619929a44d7d7170c79f4d91be1e6"}
Jun 02 15:55:10.029343 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:10.029302    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6" event={"ID":"9fe3052e-1694-446e-9f09-607c2f07f746","Type":"ContainerStarted","Data":"f56d2c41d6a6789666f97a96813b3a2bde3405b14dc4768f13ca5bb2e27e0370"}
Jun 02 15:55:10.029817 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:10.029475    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:10.049880 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:10.049829    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6" podStartSLOduration=2.288429186 podStartE2EDuration="5.049816013s" podCreationTimestamp="2026-06-02 15:55:05 +0000 UTC" firstStartedPulling="2026-06-02 15:55:06.367448255 +0000 UTC m=+493.562227733" lastFinishedPulling="2026-06-02 15:55:09.128835081 +0000 UTC m=+496.323614560" observedRunningTime="2026-06-02 15:55:10.048021221 +0000 UTC m=+497.242800723" watchObservedRunningTime="2026-06-02 15:55:10.049816013 +0000 UTC m=+497.244595513"
Jun 02 15:55:21.033896 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:21.033862    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74c68556d6-bshb6"
Jun 02 15:55:50.030494 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.030418    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"]
Jun 02 15:55:50.037489 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.037471    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:50.044173 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.044054    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jun 02 15:55:50.044265 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.044198    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jun 02 15:55:50.044502 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.044484    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kgzcl\""
Jun 02 15:55:50.066467 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.066432    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"]
Jun 02 15:55:50.088872 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.088840    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhlvx\" (UniqueName: \"kubernetes.io/projected/8a6ae02a-3982-4eaa-9a42-8eda66b28778-kube-api-access-jhlvx\") pod \"kuadrant-operator-controller-manager-57dd85bb4-hvtgk\" (UID: \"8a6ae02a-3982-4eaa-9a42-8eda66b28778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:50.089012 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.088889    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a6ae02a-3982-4eaa-9a42-8eda66b28778-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-hvtgk\" (UID: \"8a6ae02a-3982-4eaa-9a42-8eda66b28778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:50.190092 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.190045    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a6ae02a-3982-4eaa-9a42-8eda66b28778-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-hvtgk\" (UID: \"8a6ae02a-3982-4eaa-9a42-8eda66b28778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:50.190305 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.190197    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jhlvx\" (UniqueName: \"kubernetes.io/projected/8a6ae02a-3982-4eaa-9a42-8eda66b28778-kube-api-access-jhlvx\") pod \"kuadrant-operator-controller-manager-57dd85bb4-hvtgk\" (UID: \"8a6ae02a-3982-4eaa-9a42-8eda66b28778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:50.190502 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.190478    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a6ae02a-3982-4eaa-9a42-8eda66b28778-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-hvtgk\" (UID: \"8a6ae02a-3982-4eaa-9a42-8eda66b28778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:50.206202 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.206153    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhlvx\" (UniqueName: \"kubernetes.io/projected/8a6ae02a-3982-4eaa-9a42-8eda66b28778-kube-api-access-jhlvx\") pod \"kuadrant-operator-controller-manager-57dd85bb4-hvtgk\" (UID: \"8a6ae02a-3982-4eaa-9a42-8eda66b28778\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:50.349553 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.349464    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:50.490134 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:50.490102    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"]
Jun 02 15:55:50.497610 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:55:50.497583    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6ae02a_3982_4eaa_9a42_8eda66b28778.slice/crio-2efa0ac3bd60195da2342e18c72352e51e5699776c64f44268f6649f6d66bc03 WatchSource:0}: Error finding container 2efa0ac3bd60195da2342e18c72352e51e5699776c64f44268f6649f6d66bc03: Status 404 returned error can't find the container with id 2efa0ac3bd60195da2342e18c72352e51e5699776c64f44268f6649f6d66bc03
Jun 02 15:55:51.157571 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:51.157533    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk" event={"ID":"8a6ae02a-3982-4eaa-9a42-8eda66b28778","Type":"ContainerStarted","Data":"2efa0ac3bd60195da2342e18c72352e51e5699776c64f44268f6649f6d66bc03"}
Jun 02 15:55:56.176505 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:56.176463    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk" event={"ID":"8a6ae02a-3982-4eaa-9a42-8eda66b28778","Type":"ContainerStarted","Data":"3baebe82a4be3e11e403df7fde368671825d54a9cff7a8931016ab75d366b578"}
Jun 02 15:55:56.176898 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:56.176570    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:55:56.210122 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:55:56.210072    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk" podStartSLOduration=2.320079821 podStartE2EDuration="7.210055681s" podCreationTimestamp="2026-06-02 15:55:49 +0000 UTC" firstStartedPulling="2026-06-02 15:55:50.500934472 +0000 UTC m=+537.695713954" lastFinishedPulling="2026-06-02 15:55:55.390910331 +0000 UTC m=+542.585689814" observedRunningTime="2026-06-02 15:55:56.208363802 +0000 UTC m=+543.403143303" watchObservedRunningTime="2026-06-02 15:55:56.210055681 +0000 UTC m=+543.404835225"
Jun 02 15:56:07.182631 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:07.182602    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-hvtgk"
Jun 02 15:56:41.206759 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.206722    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-rmp8h"]
Jun 02 15:56:41.209203 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.209185    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:41.212273 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.212253    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jun 02 15:56:41.212388 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.212313    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-989ds\""
Jun 02 15:56:41.222103 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.222080    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-rmp8h"]
Jun 02 15:56:41.246037 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.246013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e6e797a-89ac-4ab6-8631-60ee75166155-config-file\") pod \"limitador-limitador-67566c68b4-rmp8h\" (UID: \"4e6e797a-89ac-4ab6-8631-60ee75166155\") " pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:41.246183 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.246042    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nfgh\" (UniqueName: \"kubernetes.io/projected/4e6e797a-89ac-4ab6-8631-60ee75166155-kube-api-access-2nfgh\") pod \"limitador-limitador-67566c68b4-rmp8h\" (UID: \"4e6e797a-89ac-4ab6-8631-60ee75166155\") " pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:41.253975 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.253952    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-rmp8h"]
Jun 02 15:56:41.347335 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.347297    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e6e797a-89ac-4ab6-8631-60ee75166155-config-file\") pod \"limitador-limitador-67566c68b4-rmp8h\" (UID: \"4e6e797a-89ac-4ab6-8631-60ee75166155\") " pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:41.347335 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.347339    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2nfgh\" (UniqueName: \"kubernetes.io/projected/4e6e797a-89ac-4ab6-8631-60ee75166155-kube-api-access-2nfgh\") pod \"limitador-limitador-67566c68b4-rmp8h\" (UID: \"4e6e797a-89ac-4ab6-8631-60ee75166155\") " pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:41.347930 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.347907    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e6e797a-89ac-4ab6-8631-60ee75166155-config-file\") pod \"limitador-limitador-67566c68b4-rmp8h\" (UID: \"4e6e797a-89ac-4ab6-8631-60ee75166155\") " pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:41.357012 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.356986    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nfgh\" (UniqueName: \"kubernetes.io/projected/4e6e797a-89ac-4ab6-8631-60ee75166155-kube-api-access-2nfgh\") pod \"limitador-limitador-67566c68b4-rmp8h\" (UID: \"4e6e797a-89ac-4ab6-8631-60ee75166155\") " pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:41.519457 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.519366    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:41.651014 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:41.650874    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-67566c68b4-rmp8h"]
Jun 02 15:56:41.653881 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:56:41.653848    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6e797a_89ac_4ab6_8631_60ee75166155.slice/crio-d48504b59df901ffd668713cd25059d6d434819af1f96a496d7da15357e6b016 WatchSource:0}: Error finding container d48504b59df901ffd668713cd25059d6d434819af1f96a496d7da15357e6b016: Status 404 returned error can't find the container with id d48504b59df901ffd668713cd25059d6d434819af1f96a496d7da15357e6b016
Jun 02 15:56:42.329207 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:42.329149    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h" event={"ID":"4e6e797a-89ac-4ab6-8631-60ee75166155","Type":"ContainerStarted","Data":"d48504b59df901ffd668713cd25059d6d434819af1f96a496d7da15357e6b016"}
Jun 02 15:56:47.347724 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:47.347689    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h" event={"ID":"4e6e797a-89ac-4ab6-8631-60ee75166155","Type":"ContainerStarted","Data":"5098da96527d50b90151d47b6b02a74ff0695e2ed39ff59ce605dc5567d56318"}
Jun 02 15:56:47.348138 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:47.347745    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:56:47.374598 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:47.374544    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h" podStartSLOduration=0.850405446 podStartE2EDuration="6.374528977s" podCreationTimestamp="2026-06-02 15:56:41 +0000 UTC" firstStartedPulling="2026-06-02 15:56:41.65551625 +0000 UTC m=+588.850295729" lastFinishedPulling="2026-06-02 15:56:47.179639783 +0000 UTC m=+594.374419260" observedRunningTime="2026-06-02 15:56:47.371878459 +0000 UTC m=+594.566657960" watchObservedRunningTime="2026-06-02 15:56:47.374528977 +0000 UTC m=+594.569308532"
Jun 02 15:56:53.344346 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:53.344309    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 15:56:53.345119 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:53.345099    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 15:56:53.350217 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:53.350197    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 15:56:53.351331 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:53.351310    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 15:56:58.352913 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:56:58.352880    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-67566c68b4-rmp8h"
Jun 02 15:59:49.704978 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.704942    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"]
Jun 02 15:59:49.707391 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.707368    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.707887 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.707861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.707966 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.707917    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.708035 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.708019    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-model-cache\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.708071 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.708049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-home\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.708115 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.708070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-dshm\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.708217 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.708200    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.708281 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.708243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7q56\" (UniqueName: \"kubernetes.io/projected/f94dd6ca-62b9-49e4-80d7-d567916b270a-kube-api-access-t7q56\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.710475 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.710459    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 15:59:49.711627 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.711610    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 15:59:49.711724 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.711624    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 15:59:49.711777 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.711764    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jun 02 15:59:49.721372 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.721349    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"]
Jun 02 15:59:49.809334 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809299    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809518 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809340    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t7q56\" (UniqueName: \"kubernetes.io/projected/f94dd6ca-62b9-49e4-80d7-d567916b270a-kube-api-access-t7q56\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809518 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809370    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809518 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809395    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809518 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809448    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-model-cache\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809518 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809480    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-home\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809518 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809507    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-dshm\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809850 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809724    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-tmp-dir\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809850 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809811    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809957 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809903    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-model-cache\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.809957 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.809922    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-home\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.811824 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.811803    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-dshm\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.811961 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.811943    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:49.820134 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:49.820104    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7q56\" (UniqueName: \"kubernetes.io/projected/f94dd6ca-62b9-49e4-80d7-d567916b270a-kube-api-access-t7q56\") pod \"scheduler-inline-config-test-kserve-f98c474ff-fcr46\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:50.017141 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:50.017049    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 15:59:50.182418 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:50.182390    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"]
Jun 02 15:59:50.184394 ip-10-0-135-231 kubenswrapper[2561]: W0602 15:59:50.184363    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94dd6ca_62b9_49e4_80d7_d567916b270a.slice/crio-16d00cfb1bfe7bef18cfabfc795228e581d86fabd15fd79b57a696a5de1b6d2d WatchSource:0}: Error finding container 16d00cfb1bfe7bef18cfabfc795228e581d86fabd15fd79b57a696a5de1b6d2d: Status 404 returned error can't find the container with id 16d00cfb1bfe7bef18cfabfc795228e581d86fabd15fd79b57a696a5de1b6d2d
Jun 02 15:59:50.186077 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:50.186057    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:59:50.956199 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:50.956139    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46" event={"ID":"f94dd6ca-62b9-49e4-80d7-d567916b270a","Type":"ContainerStarted","Data":"16d00cfb1bfe7bef18cfabfc795228e581d86fabd15fd79b57a696a5de1b6d2d"}
Jun 02 15:59:53.968369 ip-10-0-135-231 kubenswrapper[2561]: I0602 15:59:53.968331    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46" event={"ID":"f94dd6ca-62b9-49e4-80d7-d567916b270a","Type":"ContainerStarted","Data":"707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd"}
Jun 02 16:00:00.530668 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.530637    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"]
Jun 02 16:00:00.535539 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.535518    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.543226 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.543207    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jun 02 16:00:00.555883 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.555852    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"]
Jun 02 16:00:00.600144 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.600115    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.600310 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.600149    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.600310 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.600202    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.600310 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.600226    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5dp\" (UniqueName: \"kubernetes.io/projected/b8ef73fb-952b-4e4e-8827-b561850b6f66-kube-api-access-8f5dp\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.600310 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.600258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ef73fb-952b-4e4e-8827-b561850b6f66-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.600502 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.600305    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.600502 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.600364    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701081 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701047    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ef73fb-952b-4e4e-8827-b561850b6f66-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701081 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701083    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701331 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701109    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701331 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701154    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701331 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701204    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701331 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701266    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701331 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701306    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5dp\" (UniqueName: \"kubernetes.io/projected/b8ef73fb-952b-4e4e-8827-b561850b6f66-kube-api-access-8f5dp\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701584 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701518    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701639 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701585    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701683 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701647    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-kserve-provision-location\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.701760 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.701738    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.703420 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.703396    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.703747 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.703729    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ef73fb-952b-4e4e-8827-b561850b6f66-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.711520 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.711499    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5dp\" (UniqueName: \"kubernetes.io/projected/b8ef73fb-952b-4e4e-8827-b561850b6f66-kube-api-access-8f5dp\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.846019 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.845936    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:00.983639 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.983609    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"]
Jun 02 16:00:00.987017 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:00:00.986991    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ef73fb_952b_4e4e_8827_b561850b6f66.slice/crio-d9c09f54d30cf72d1f01307d7ede580040b8ea3c40561018e270b46dffb64c88 WatchSource:0}: Error finding container d9c09f54d30cf72d1f01307d7ede580040b8ea3c40561018e270b46dffb64c88: Status 404 returned error can't find the container with id d9c09f54d30cf72d1f01307d7ede580040b8ea3c40561018e270b46dffb64c88
Jun 02 16:00:00.996327 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:00.996298    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t" event={"ID":"b8ef73fb-952b-4e4e-8827-b561850b6f66","Type":"ContainerStarted","Data":"d9c09f54d30cf72d1f01307d7ede580040b8ea3c40561018e270b46dffb64c88"}
Jun 02 16:00:02.001740 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:02.001696    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t" event={"ID":"b8ef73fb-952b-4e4e-8827-b561850b6f66","Type":"ContainerStarted","Data":"5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e"}
Jun 02 16:00:14.359806 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:14.359728    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"]
Jun 02 16:00:14.360269 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:14.360015    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t" podUID="b8ef73fb-952b-4e4e-8827-b561850b6f66" containerName="storage-initializer" containerID="cri-o://5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e" gracePeriod=30
Jun 02 16:00:44.547175 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.547133    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t_b8ef73fb-952b-4e4e-8827-b561850b6f66/storage-initializer/0.log"
Jun 02 16:00:44.547471 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.547221    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:44.608132 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608096    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ef73fb-952b-4e4e-8827-b561850b6f66-tls-certs\") pod \"b8ef73fb-952b-4e4e-8827-b561850b6f66\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") "
Jun 02 16:00:44.608132 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608141    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-dshm\") pod \"b8ef73fb-952b-4e4e-8827-b561850b6f66\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") "
Jun 02 16:00:44.608354 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608194    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-home\") pod \"b8ef73fb-952b-4e4e-8827-b561850b6f66\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") "
Jun 02 16:00:44.608354 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608228    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-kserve-provision-location\") pod \"b8ef73fb-952b-4e4e-8827-b561850b6f66\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") "
Jun 02 16:00:44.608354 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608253    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-model-cache\") pod \"b8ef73fb-952b-4e4e-8827-b561850b6f66\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") "
Jun 02 16:00:44.608354 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608325    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f5dp\" (UniqueName: \"kubernetes.io/projected/b8ef73fb-952b-4e4e-8827-b561850b6f66-kube-api-access-8f5dp\") pod \"b8ef73fb-952b-4e4e-8827-b561850b6f66\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") "
Jun 02 16:00:44.608596 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608367    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-tmp-dir\") pod \"b8ef73fb-952b-4e4e-8827-b561850b6f66\" (UID: \"b8ef73fb-952b-4e4e-8827-b561850b6f66\") "
Jun 02 16:00:44.608596 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608508    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-home" (OuterVolumeSpecName: "home") pod "b8ef73fb-952b-4e4e-8827-b561850b6f66" (UID: "b8ef73fb-952b-4e4e-8827-b561850b6f66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:44.608596 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608556    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8ef73fb-952b-4e4e-8827-b561850b6f66" (UID: "b8ef73fb-952b-4e4e-8827-b561850b6f66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:44.608748 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608688    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:00:44.608748 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608706    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:00:44.608748 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.608730    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8ef73fb-952b-4e4e-8827-b561850b6f66" (UID: "b8ef73fb-952b-4e4e-8827-b561850b6f66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:44.610338 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.610313    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-dshm" (OuterVolumeSpecName: "dshm") pod "b8ef73fb-952b-4e4e-8827-b561850b6f66" (UID: "b8ef73fb-952b-4e4e-8827-b561850b6f66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:44.610735 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.610715    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ef73fb-952b-4e4e-8827-b561850b6f66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8ef73fb-952b-4e4e-8827-b561850b6f66" (UID: "b8ef73fb-952b-4e4e-8827-b561850b6f66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:00:44.610825 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.610759    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ef73fb-952b-4e4e-8827-b561850b6f66-kube-api-access-8f5dp" (OuterVolumeSpecName: "kube-api-access-8f5dp") pod "b8ef73fb-952b-4e4e-8827-b561850b6f66" (UID: "b8ef73fb-952b-4e4e-8827-b561850b6f66"). InnerVolumeSpecName "kube-api-access-8f5dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:00:44.668663 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.668595    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8ef73fb-952b-4e4e-8827-b561850b6f66" (UID: "b8ef73fb-952b-4e4e-8827-b561850b6f66"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:00:44.710150 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.710114    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:00:44.710150 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.710147    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ef73fb-952b-4e4e-8827-b561850b6f66-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:00:44.710341 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.710180    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:00:44.710341 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.710196    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ef73fb-952b-4e4e-8827-b561850b6f66-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:00:44.710341 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:44.710208    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f5dp\" (UniqueName: \"kubernetes.io/projected/b8ef73fb-952b-4e4e-8827-b561850b6f66-kube-api-access-8f5dp\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:00:45.145869 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.145840    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t_b8ef73fb-952b-4e4e-8827-b561850b6f66/storage-initializer/0.log"
Jun 02 16:00:45.146109 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.145881    2561 generic.go:358] "Generic (PLEG): container finished" podID="b8ef73fb-952b-4e4e-8827-b561850b6f66" containerID="5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e" exitCode=137
Jun 02 16:00:45.146109 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.145958    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"
Jun 02 16:00:45.146109 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.145969    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t" event={"ID":"b8ef73fb-952b-4e4e-8827-b561850b6f66","Type":"ContainerDied","Data":"5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e"}
Jun 02 16:00:45.146109 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.146008    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t" event={"ID":"b8ef73fb-952b-4e4e-8827-b561850b6f66","Type":"ContainerDied","Data":"d9c09f54d30cf72d1f01307d7ede580040b8ea3c40561018e270b46dffb64c88"}
Jun 02 16:00:45.146109 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.146028    2561 scope.go:117] "RemoveContainer" containerID="5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e"
Jun 02 16:00:45.190032 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.190002    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"]
Jun 02 16:00:45.195731 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.195700    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-f1d92d0f-kserve-7d7f7968c-q476t"]
Jun 02 16:00:45.216492 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.216468    2561 scope.go:117] "RemoveContainer" containerID="5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e"
Jun 02 16:00:45.216796 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:00:45.216771    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e\": container with ID starting with 5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e not found: ID does not exist" containerID="5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e"
Jun 02 16:00:45.216871 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.216805    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e"} err="failed to get container status \"5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e\": rpc error: code = NotFound desc = could not find container \"5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e\": container with ID starting with 5da84a128e354ded5236626ab921ea9586811d54dde722ab97f1ec6229a5fc2e not found: ID does not exist"
Jun 02 16:00:45.433092 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:45.433012    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ef73fb-952b-4e4e-8827-b561850b6f66" path="/var/lib/kubelet/pods/b8ef73fb-952b-4e4e-8827-b561850b6f66/volumes"
Jun 02 16:00:49.161095 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:49.161057    2561 generic.go:358] "Generic (PLEG): container finished" podID="f94dd6ca-62b9-49e4-80d7-d567916b270a" containerID="707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd" exitCode=0
Jun 02 16:00:49.161514 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:00:49.161130    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46" event={"ID":"f94dd6ca-62b9-49e4-80d7-d567916b270a","Type":"ContainerDied","Data":"707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd"}
Jun 02 16:01:09.233830 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:09.233799    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46" event={"ID":"f94dd6ca-62b9-49e4-80d7-d567916b270a","Type":"ContainerStarted","Data":"376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4"}
Jun 02 16:01:09.254799 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:09.254730    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46" podStartSLOduration=1.72543164 podStartE2EDuration="1m20.254711213s" podCreationTimestamp="2026-06-02 15:59:49 +0000 UTC" firstStartedPulling="2026-06-02 15:59:50.18620672 +0000 UTC m=+777.380986199" lastFinishedPulling="2026-06-02 16:01:08.715486294 +0000 UTC m=+855.910265772" observedRunningTime="2026-06-02 16:01:09.253461291 +0000 UTC m=+856.448240795" watchObservedRunningTime="2026-06-02 16:01:09.254711213 +0000 UTC m=+856.449490714"
Jun 02 16:01:10.017342 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:10.017306    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 16:01:10.017520 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:10.017372    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 16:01:10.030112 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:10.030084    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 16:01:10.248979 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:10.248950    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 16:01:11.233209 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:11.233176    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"]
Jun 02 16:01:11.353030 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:01:11.352994    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs: secret "scheduler-inline-config-test-kserve-self-signed-certs" not found
Jun 02 16:01:11.353446 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:01:11.353068    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs podName:f94dd6ca-62b9-49e4-80d7-d567916b270a nodeName:}" failed. No retries permitted until 2026-06-02 16:01:11.853048232 +0000 UTC m=+859.047827710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs") pod "scheduler-inline-config-test-kserve-f98c474ff-fcr46" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a") : secret "scheduler-inline-config-test-kserve-self-signed-certs" not found
Jun 02 16:01:11.858092 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:01:11.858057    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/scheduler-inline-config-test-kserve-self-signed-certs: secret "scheduler-inline-config-test-kserve-self-signed-certs" not found
Jun 02 16:01:11.858307 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:01:11.858143    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs podName:f94dd6ca-62b9-49e4-80d7-d567916b270a nodeName:}" failed. No retries permitted until 2026-06-02 16:01:12.858119542 +0000 UTC m=+860.052899020 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs") pod "scheduler-inline-config-test-kserve-f98c474ff-fcr46" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a") : secret "scheduler-inline-config-test-kserve-self-signed-certs" not found
Jun 02 16:01:12.245490 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.245448    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46" podUID="f94dd6ca-62b9-49e4-80d7-d567916b270a" containerName="main" containerID="cri-o://376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4" gracePeriod=30
Jun 02 16:01:12.512208 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.512184    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 16:01:12.664924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.664888    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-home\") pod \"f94dd6ca-62b9-49e4-80d7-d567916b270a\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") "
Jun 02 16:01:12.664924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.664926    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7q56\" (UniqueName: \"kubernetes.io/projected/f94dd6ca-62b9-49e4-80d7-d567916b270a-kube-api-access-t7q56\") pod \"f94dd6ca-62b9-49e4-80d7-d567916b270a\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") "
Jun 02 16:01:12.665194 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.664957    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-tmp-dir\") pod \"f94dd6ca-62b9-49e4-80d7-d567916b270a\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") "
Jun 02 16:01:12.665194 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.664988    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-kserve-provision-location\") pod \"f94dd6ca-62b9-49e4-80d7-d567916b270a\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") "
Jun 02 16:01:12.665194 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665022    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-model-cache\") pod \"f94dd6ca-62b9-49e4-80d7-d567916b270a\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") "
Jun 02 16:01:12.665194 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665038    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-dshm\") pod \"f94dd6ca-62b9-49e4-80d7-d567916b270a\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") "
Jun 02 16:01:12.665194 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665091    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs\") pod \"f94dd6ca-62b9-49e4-80d7-d567916b270a\" (UID: \"f94dd6ca-62b9-49e4-80d7-d567916b270a\") "
Jun 02 16:01:12.665194 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665145    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-home" (OuterVolumeSpecName: "home") pod "f94dd6ca-62b9-49e4-80d7-d567916b270a" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:12.665475 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665291    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-model-cache" (OuterVolumeSpecName: "model-cache") pod "f94dd6ca-62b9-49e4-80d7-d567916b270a" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:12.665475 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665326    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f94dd6ca-62b9-49e4-80d7-d567916b270a" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:12.665475 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665401    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:12.665475 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665426    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:12.665475 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.665441    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:12.667237 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.667143    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94dd6ca-62b9-49e4-80d7-d567916b270a-kube-api-access-t7q56" (OuterVolumeSpecName: "kube-api-access-t7q56") pod "f94dd6ca-62b9-49e4-80d7-d567916b270a" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a"). InnerVolumeSpecName "kube-api-access-t7q56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:01:12.667237 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.667186    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-dshm" (OuterVolumeSpecName: "dshm") pod "f94dd6ca-62b9-49e4-80d7-d567916b270a" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:12.667384 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.667192    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f94dd6ca-62b9-49e4-80d7-d567916b270a" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:01:12.727377 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.727329    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f94dd6ca-62b9-49e4-80d7-d567916b270a" (UID: "f94dd6ca-62b9-49e4-80d7-d567916b270a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:12.765964 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.765882    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:12.765964 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.765920    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f94dd6ca-62b9-49e4-80d7-d567916b270a-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:12.765964 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.765931    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f94dd6ca-62b9-49e4-80d7-d567916b270a-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:12.765964 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:12.765942    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7q56\" (UniqueName: \"kubernetes.io/projected/f94dd6ca-62b9-49e4-80d7-d567916b270a-kube-api-access-t7q56\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:13.250330 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.250293    2561 generic.go:358] "Generic (PLEG): container finished" podID="f94dd6ca-62b9-49e4-80d7-d567916b270a" containerID="376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4" exitCode=0
Jun 02 16:01:13.250512 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.250350    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46" event={"ID":"f94dd6ca-62b9-49e4-80d7-d567916b270a","Type":"ContainerDied","Data":"376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4"}
Jun 02 16:01:13.250512 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.250372    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46" event={"ID":"f94dd6ca-62b9-49e4-80d7-d567916b270a","Type":"ContainerDied","Data":"16d00cfb1bfe7bef18cfabfc795228e581d86fabd15fd79b57a696a5de1b6d2d"}
Jun 02 16:01:13.250512 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.250374    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"
Jun 02 16:01:13.250512 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.250386    2561 scope.go:117] "RemoveContainer" containerID="376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4"
Jun 02 16:01:13.258902 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.258886    2561 scope.go:117] "RemoveContainer" containerID="707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd"
Jun 02 16:01:13.273860 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.273835    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"]
Jun 02 16:01:13.286027 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.286000    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-f98c474ff-fcr46"]
Jun 02 16:01:13.321027 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.321007    2561 scope.go:117] "RemoveContainer" containerID="376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4"
Jun 02 16:01:13.321374 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:01:13.321352    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4\": container with ID starting with 376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4 not found: ID does not exist" containerID="376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4"
Jun 02 16:01:13.321443 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.321384    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4"} err="failed to get container status \"376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4\": rpc error: code = NotFound desc = could not find container \"376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4\": container with ID starting with 376e853881a6ac80b701db6aa71ec51ce5f4d1f94364d562b21bc6f172948fb4 not found: ID does not exist"
Jun 02 16:01:13.321443 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.321405    2561 scope.go:117] "RemoveContainer" containerID="707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd"
Jun 02 16:01:13.321678 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:01:13.321659    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd\": container with ID starting with 707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd not found: ID does not exist" containerID="707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd"
Jun 02 16:01:13.321727 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.321683    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd"} err="failed to get container status \"707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd\": rpc error: code = NotFound desc = could not find container \"707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd\": container with ID starting with 707b11aecd5d9c5af93df6230ae628f6888ccfc871d9160f3a92e763acccd8bd not found: ID does not exist"
Jun 02 16:01:13.433794 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:13.433759    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f94dd6ca-62b9-49e4-80d7-d567916b270a" path="/var/lib/kubelet/pods/f94dd6ca-62b9-49e4-80d7-d567916b270a/volumes"
Jun 02 16:01:17.330282 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330246    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"]
Jun 02 16:01:17.330836 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330739    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94dd6ca-62b9-49e4-80d7-d567916b270a" containerName="storage-initializer"
Jun 02 16:01:17.330836 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330758    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94dd6ca-62b9-49e4-80d7-d567916b270a" containerName="storage-initializer"
Jun 02 16:01:17.330836 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330780    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ef73fb-952b-4e4e-8827-b561850b6f66" containerName="storage-initializer"
Jun 02 16:01:17.330836 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330788    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ef73fb-952b-4e4e-8827-b561850b6f66" containerName="storage-initializer"
Jun 02 16:01:17.330836 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330801    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f94dd6ca-62b9-49e4-80d7-d567916b270a" containerName="main"
Jun 02 16:01:17.330836 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330811    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f94dd6ca-62b9-49e4-80d7-d567916b270a" containerName="main"
Jun 02 16:01:17.331137 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330925    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="f94dd6ca-62b9-49e4-80d7-d567916b270a" containerName="main"
Jun 02 16:01:17.331137 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.330943    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="b8ef73fb-952b-4e4e-8827-b561850b6f66" containerName="storage-initializer"
Jun 02 16:01:17.334896 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.334877    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.337913 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.337894    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jun 02 16:01:17.339304 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.339278    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 16:01:17.339399 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.339303    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:01:17.339399 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.339320    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:01:17.349429 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.349410    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"]
Jun 02 16:01:17.507030 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.506990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-dshm\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.507231 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.507047    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.507231 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.507080    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-home\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.507231 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.507193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf94411-d6d3-4954-9a17-efb14061943b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.507231 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.507220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.513503 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.507276    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzs8g\" (UniqueName: \"kubernetes.io/projected/cdf94411-d6d3-4954-9a17-efb14061943b-kube-api-access-nzs8g\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.513503 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.507910    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.608724 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.608635    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf94411-d6d3-4954-9a17-efb14061943b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.608724 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.608683    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.608943 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.608734    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nzs8g\" (UniqueName: \"kubernetes.io/projected/cdf94411-d6d3-4954-9a17-efb14061943b-kube-api-access-nzs8g\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.608943 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.608776    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.608943 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.608798    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-dshm\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.608943 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.608829    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.608943 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.608852    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-home\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.609275 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.609246    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.609365 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.609255    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-home\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.609365 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.609351    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.609683 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.609663    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.611589 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.611558    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf94411-d6d3-4954-9a17-efb14061943b-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.611690 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.611613    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-dshm\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.624972 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.624948    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzs8g\" (UniqueName: \"kubernetes.io/projected/cdf94411-d6d3-4954-9a17-efb14061943b-kube-api-access-nzs8g\") pod \"scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.645797 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.645769    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:17.776350 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:17.776316    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"]
Jun 02 16:01:17.778188 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:01:17.778150    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf94411_d6d3_4954_9a17_efb14061943b.slice/crio-d37f04227ecbcd72b5ad0e2c7295391ccee0474f6b449db438ee4d1fc50821c3 WatchSource:0}: Error finding container d37f04227ecbcd72b5ad0e2c7295391ccee0474f6b449db438ee4d1fc50821c3: Status 404 returned error can't find the container with id d37f04227ecbcd72b5ad0e2c7295391ccee0474f6b449db438ee4d1fc50821c3
Jun 02 16:01:18.269339 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:18.269303    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464" event={"ID":"cdf94411-d6d3-4954-9a17-efb14061943b","Type":"ContainerStarted","Data":"81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5"}
Jun 02 16:01:18.269339 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:18.269338    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464" event={"ID":"cdf94411-d6d3-4954-9a17-efb14061943b","Type":"ContainerStarted","Data":"d37f04227ecbcd72b5ad0e2c7295391ccee0474f6b449db438ee4d1fc50821c3"}
Jun 02 16:01:22.285373 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:22.285281    2561 generic.go:358] "Generic (PLEG): container finished" podID="cdf94411-d6d3-4954-9a17-efb14061943b" containerID="81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5" exitCode=0
Jun 02 16:01:22.285373 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:22.285359    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464" event={"ID":"cdf94411-d6d3-4954-9a17-efb14061943b","Type":"ContainerDied","Data":"81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5"}
Jun 02 16:01:23.290377 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:23.290341    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464" event={"ID":"cdf94411-d6d3-4954-9a17-efb14061943b","Type":"ContainerStarted","Data":"14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e"}
Jun 02 16:01:23.312552 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:23.312502    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464" podStartSLOduration=6.312488103 podStartE2EDuration="6.312488103s" podCreationTimestamp="2026-06-02 16:01:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:01:23.310866752 +0000 UTC m=+870.505646252" watchObservedRunningTime="2026-06-02 16:01:23.312488103 +0000 UTC m=+870.507267602"
Jun 02 16:01:27.646371 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:27.646329    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:27.646782 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:27.646388    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:27.659005 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:27.658977    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:28.319131 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:28.319099    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:50.273917 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.273831    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"]
Jun 02 16:01:50.274808 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.274178    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464" podUID="cdf94411-d6d3-4954-9a17-efb14061943b" containerName="main" containerID="cri-o://14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e" gracePeriod=30
Jun 02 16:01:50.536257 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.536202    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:50.610002 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.609966    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-home\") pod \"cdf94411-d6d3-4954-9a17-efb14061943b\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") "
Jun 02 16:01:50.610157 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610013    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzs8g\" (UniqueName: \"kubernetes.io/projected/cdf94411-d6d3-4954-9a17-efb14061943b-kube-api-access-nzs8g\") pod \"cdf94411-d6d3-4954-9a17-efb14061943b\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") "
Jun 02 16:01:50.610157 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610032    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-kserve-provision-location\") pod \"cdf94411-d6d3-4954-9a17-efb14061943b\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") "
Jun 02 16:01:50.610157 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610141    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-tmp-dir\") pod \"cdf94411-d6d3-4954-9a17-efb14061943b\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") "
Jun 02 16:01:50.610361 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610227    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf94411-d6d3-4954-9a17-efb14061943b-tls-certs\") pod \"cdf94411-d6d3-4954-9a17-efb14061943b\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") "
Jun 02 16:01:50.610361 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610254    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-model-cache\") pod \"cdf94411-d6d3-4954-9a17-efb14061943b\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") "
Jun 02 16:01:50.610361 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610301    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-dshm\") pod \"cdf94411-d6d3-4954-9a17-efb14061943b\" (UID: \"cdf94411-d6d3-4954-9a17-efb14061943b\") "
Jun 02 16:01:50.610497 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610372    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-home" (OuterVolumeSpecName: "home") pod "cdf94411-d6d3-4954-9a17-efb14061943b" (UID: "cdf94411-d6d3-4954-9a17-efb14061943b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:50.610497 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610409    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdf94411-d6d3-4954-9a17-efb14061943b" (UID: "cdf94411-d6d3-4954-9a17-efb14061943b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:50.610617 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610598    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:50.610656 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610603    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdf94411-d6d3-4954-9a17-efb14061943b" (UID: "cdf94411-d6d3-4954-9a17-efb14061943b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:50.610656 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.610623    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:50.612254 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.612229    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdf94411-d6d3-4954-9a17-efb14061943b-kube-api-access-nzs8g" (OuterVolumeSpecName: "kube-api-access-nzs8g") pod "cdf94411-d6d3-4954-9a17-efb14061943b" (UID: "cdf94411-d6d3-4954-9a17-efb14061943b"). InnerVolumeSpecName "kube-api-access-nzs8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:01:50.612355 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.612321    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-dshm" (OuterVolumeSpecName: "dshm") pod "cdf94411-d6d3-4954-9a17-efb14061943b" (UID: "cdf94411-d6d3-4954-9a17-efb14061943b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:50.612570 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.612543    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdf94411-d6d3-4954-9a17-efb14061943b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdf94411-d6d3-4954-9a17-efb14061943b" (UID: "cdf94411-d6d3-4954-9a17-efb14061943b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:01:50.664590 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.664546    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdf94411-d6d3-4954-9a17-efb14061943b" (UID: "cdf94411-d6d3-4954-9a17-efb14061943b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:01:50.711736 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.711705    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:50.711736 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.711738    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzs8g\" (UniqueName: \"kubernetes.io/projected/cdf94411-d6d3-4954-9a17-efb14061943b-kube-api-access-nzs8g\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:50.711961 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.711751    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:50.711961 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.711761    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdf94411-d6d3-4954-9a17-efb14061943b-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:50.711961 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:50.711770    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdf94411-d6d3-4954-9a17-efb14061943b-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:01:51.383382 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.383352    2561 generic.go:358] "Generic (PLEG): container finished" podID="cdf94411-d6d3-4954-9a17-efb14061943b" containerID="14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e" exitCode=0
Jun 02 16:01:51.383765 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.383431    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"
Jun 02 16:01:51.383765 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.383441    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464" event={"ID":"cdf94411-d6d3-4954-9a17-efb14061943b","Type":"ContainerDied","Data":"14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e"}
Jun 02 16:01:51.383765 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.383479    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464" event={"ID":"cdf94411-d6d3-4954-9a17-efb14061943b","Type":"ContainerDied","Data":"d37f04227ecbcd72b5ad0e2c7295391ccee0474f6b449db438ee4d1fc50821c3"}
Jun 02 16:01:51.383765 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.383499    2561 scope.go:117] "RemoveContainer" containerID="14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e"
Jun 02 16:01:51.391655 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.391636    2561 scope.go:117] "RemoveContainer" containerID="81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5"
Jun 02 16:01:51.405919 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.405887    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"]
Jun 02 16:01:51.406927 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.406912    2561 scope.go:117] "RemoveContainer" containerID="14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e"
Jun 02 16:01:51.407212 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:01:51.407192    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e\": container with ID starting with 14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e not found: ID does not exist" containerID="14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e"
Jun 02 16:01:51.407303 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.407219    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e"} err="failed to get container status \"14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e\": rpc error: code = NotFound desc = could not find container \"14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e\": container with ID starting with 14bc0d741a75fdb73731982f4e8c422c0b1788baf59fa338b024a00b42d5786e not found: ID does not exist"
Jun 02 16:01:51.407303 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.407239    2561 scope.go:117] "RemoveContainer" containerID="81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5"
Jun 02 16:01:51.407532 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:01:51.407505    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5\": container with ID starting with 81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5 not found: ID does not exist" containerID="81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5"
Jun 02 16:01:51.407603 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.407539    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5"} err="failed to get container status \"81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5\": rpc error: code = NotFound desc = could not find container \"81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5\": container with ID starting with 81643f4dd3d4f7db544ce1208a9cc25eadab732d2cc4f4aa8ada34596b46c4f5 not found: ID does not exist"
Jun 02 16:01:51.408272 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.408255    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-5cd84c95dc-c8464"]
Jun 02 16:01:51.433728 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:51.433697    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdf94411-d6d3-4954-9a17-efb14061943b" path="/var/lib/kubelet/pods/cdf94411-d6d3-4954-9a17-efb14061943b/volumes"
Jun 02 16:01:53.368424 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:53.368394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:01:53.370482 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:53.370463    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:01:53.374530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:53.374510    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:01:53.375989 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:53.375973    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:01:59.057545 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.057517    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"]
Jun 02 16:01:59.057896 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.057851    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf94411-d6d3-4954-9a17-efb14061943b" containerName="main"
Jun 02 16:01:59.057896 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.057863    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf94411-d6d3-4954-9a17-efb14061943b" containerName="main"
Jun 02 16:01:59.057896 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.057885    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdf94411-d6d3-4954-9a17-efb14061943b" containerName="storage-initializer"
Jun 02 16:01:59.057896 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.057892    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdf94411-d6d3-4954-9a17-efb14061943b" containerName="storage-initializer"
Jun 02 16:01:59.058053 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.057941    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="cdf94411-d6d3-4954-9a17-efb14061943b" containerName="main"
Jun 02 16:01:59.061188 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.061149    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.064697 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.064672    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 16:01:59.065120 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.065096    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:01:59.066004 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.065941    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jun 02 16:01:59.066004 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.065949    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:01:59.078344 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.078320    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"]
Jun 02 16:01:59.187089 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.187057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-home\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.187273 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.187099    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.187273 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.187136    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6qf\" (UniqueName: \"kubernetes.io/projected/b0e9efe9-51af-494f-ad97-d510698952e3-kube-api-access-ts6qf\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.187273 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.187250    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e9efe9-51af-494f-ad97-d510698952e3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.187375 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.187301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-dshm\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.187375 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.187321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.187375 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.187367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-model-cache\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.287983 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.287944    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6qf\" (UniqueName: \"kubernetes.io/projected/b0e9efe9-51af-494f-ad97-d510698952e3-kube-api-access-ts6qf\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288139 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288012    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e9efe9-51af-494f-ad97-d510698952e3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288139 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288060    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-dshm\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288139 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288087    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288345 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288207    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-model-cache\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288345 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288273    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-home\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288345 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288325    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288510 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288492    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288579 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288561    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-model-cache\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288664 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288646    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.288699 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.288661    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-home\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.290826 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.290801    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-dshm\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.291026 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.291009    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e9efe9-51af-494f-ad97-d510698952e3-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.307488 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.307465    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6qf\" (UniqueName: \"kubernetes.io/projected/b0e9efe9-51af-494f-ad97-d510698952e3-kube-api-access-ts6qf\") pod \"scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.372030 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.371939    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:01:59.504927 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:01:59.504815    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"]
Jun 02 16:01:59.507613 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:01:59.507582    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0e9efe9_51af_494f_ad97_d510698952e3.slice/crio-594c903b9205c0f4221bfc8d681aabc3359649dd90f86ef6d83a53fd8e47a6c5 WatchSource:0}: Error finding container 594c903b9205c0f4221bfc8d681aabc3359649dd90f86ef6d83a53fd8e47a6c5: Status 404 returned error can't find the container with id 594c903b9205c0f4221bfc8d681aabc3359649dd90f86ef6d83a53fd8e47a6c5
Jun 02 16:02:00.416505 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:00.416470    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc" event={"ID":"b0e9efe9-51af-494f-ad97-d510698952e3","Type":"ContainerStarted","Data":"4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219"}
Jun 02 16:02:00.416505 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:00.416513    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc" event={"ID":"b0e9efe9-51af-494f-ad97-d510698952e3","Type":"ContainerStarted","Data":"594c903b9205c0f4221bfc8d681aabc3359649dd90f86ef6d83a53fd8e47a6c5"}
Jun 02 16:02:03.427847 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:03.427817    2561 generic.go:358] "Generic (PLEG): container finished" podID="b0e9efe9-51af-494f-ad97-d510698952e3" containerID="4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219" exitCode=0
Jun 02 16:02:03.428118 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:03.427886    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc" event={"ID":"b0e9efe9-51af-494f-ad97-d510698952e3","Type":"ContainerDied","Data":"4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219"}
Jun 02 16:02:04.432913 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:04.432879    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc" event={"ID":"b0e9efe9-51af-494f-ad97-d510698952e3","Type":"ContainerStarted","Data":"38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2"}
Jun 02 16:02:04.453401 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:04.453340    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc" podStartSLOduration=5.453319839 podStartE2EDuration="5.453319839s" podCreationTimestamp="2026-06-02 16:01:59 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:02:04.452180244 +0000 UTC m=+911.646959737" watchObservedRunningTime="2026-06-02 16:02:04.453319839 +0000 UTC m=+911.648099340"
Jun 02 16:02:09.372529 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:09.372488    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:02:09.372529 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:09.372533    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:02:09.385247 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:09.385220    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:02:09.459661 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:09.459630    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:02:33.029119 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.029085    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"]
Jun 02 16:02:33.030625 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.029397    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc" podUID="b0e9efe9-51af-494f-ad97-d510698952e3" containerName="main" containerID="cri-o://38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2" gracePeriod=30
Jun 02 16:02:33.278209 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.278188    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:02:33.397179 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397139    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts6qf\" (UniqueName: \"kubernetes.io/projected/b0e9efe9-51af-494f-ad97-d510698952e3-kube-api-access-ts6qf\") pod \"b0e9efe9-51af-494f-ad97-d510698952e3\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") "
Jun 02 16:02:33.397363 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397202    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-model-cache\") pod \"b0e9efe9-51af-494f-ad97-d510698952e3\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") "
Jun 02 16:02:33.397363 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397291    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-home\") pod \"b0e9efe9-51af-494f-ad97-d510698952e3\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") "
Jun 02 16:02:33.397363 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397329    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-kserve-provision-location\") pod \"b0e9efe9-51af-494f-ad97-d510698952e3\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") "
Jun 02 16:02:33.397363 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397351    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-dshm\") pod \"b0e9efe9-51af-494f-ad97-d510698952e3\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") "
Jun 02 16:02:33.397573 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397401    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-tmp-dir\") pod \"b0e9efe9-51af-494f-ad97-d510698952e3\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") "
Jun 02 16:02:33.397573 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397437    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0e9efe9-51af-494f-ad97-d510698952e3" (UID: "b0e9efe9-51af-494f-ad97-d510698952e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:02:33.397573 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397479    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e9efe9-51af-494f-ad97-d510698952e3-tls-certs\") pod \"b0e9efe9-51af-494f-ad97-d510698952e3\" (UID: \"b0e9efe9-51af-494f-ad97-d510698952e3\") "
Jun 02 16:02:33.397702 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397580    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-home" (OuterVolumeSpecName: "home") pod "b0e9efe9-51af-494f-ad97-d510698952e3" (UID: "b0e9efe9-51af-494f-ad97-d510698952e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:02:33.397702 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397652    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0e9efe9-51af-494f-ad97-d510698952e3" (UID: "b0e9efe9-51af-494f-ad97-d510698952e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:02:33.397798 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397783    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:02:33.397839 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397801    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:02:33.397839 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.397810    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:02:33.399485 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.399457    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0e9efe9-51af-494f-ad97-d510698952e3-kube-api-access-ts6qf" (OuterVolumeSpecName: "kube-api-access-ts6qf") pod "b0e9efe9-51af-494f-ad97-d510698952e3" (UID: "b0e9efe9-51af-494f-ad97-d510698952e3"). InnerVolumeSpecName "kube-api-access-ts6qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:02:33.399601 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.399490    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0e9efe9-51af-494f-ad97-d510698952e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0e9efe9-51af-494f-ad97-d510698952e3" (UID: "b0e9efe9-51af-494f-ad97-d510698952e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:02:33.400014 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.399993    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-dshm" (OuterVolumeSpecName: "dshm") pod "b0e9efe9-51af-494f-ad97-d510698952e3" (UID: "b0e9efe9-51af-494f-ad97-d510698952e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:02:33.453954 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.453919    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b0e9efe9-51af-494f-ad97-d510698952e3" (UID: "b0e9efe9-51af-494f-ad97-d510698952e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:02:33.499094 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.499068    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:02:33.499094 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.499093    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0e9efe9-51af-494f-ad97-d510698952e3-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:02:33.499276 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.499106    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0e9efe9-51af-494f-ad97-d510698952e3-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:02:33.499276 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.499116    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6qf\" (UniqueName: \"kubernetes.io/projected/b0e9efe9-51af-494f-ad97-d510698952e3-kube-api-access-ts6qf\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:02:33.525017 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.524989    2561 generic.go:358] "Generic (PLEG): container finished" podID="b0e9efe9-51af-494f-ad97-d510698952e3" containerID="38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2" exitCode=0
Jun 02 16:02:33.525155 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.525058    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"
Jun 02 16:02:33.525155 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.525064    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc" event={"ID":"b0e9efe9-51af-494f-ad97-d510698952e3","Type":"ContainerDied","Data":"38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2"}
Jun 02 16:02:33.525155 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.525118    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc" event={"ID":"b0e9efe9-51af-494f-ad97-d510698952e3","Type":"ContainerDied","Data":"594c903b9205c0f4221bfc8d681aabc3359649dd90f86ef6d83a53fd8e47a6c5"}
Jun 02 16:02:33.525302 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.525145    2561 scope.go:117] "RemoveContainer" containerID="38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2"
Jun 02 16:02:33.535476 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.535462    2561 scope.go:117] "RemoveContainer" containerID="4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219"
Jun 02 16:02:33.546812 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.546797    2561 scope.go:117] "RemoveContainer" containerID="38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2"
Jun 02 16:02:33.547029 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:02:33.547012    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2\": container with ID starting with 38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2 not found: ID does not exist" containerID="38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2"
Jun 02 16:02:33.547087 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.547035    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2"} err="failed to get container status \"38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2\": rpc error: code = NotFound desc = could not find container \"38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2\": container with ID starting with 38e0ade33002105ac495963ebb1e3276b40d84e1e18547e1249e69509f61b5d2 not found: ID does not exist"
Jun 02 16:02:33.547087 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.547050    2561 scope.go:117] "RemoveContainer" containerID="4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219"
Jun 02 16:02:33.547291 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:02:33.547270    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219\": container with ID starting with 4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219 not found: ID does not exist" containerID="4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219"
Jun 02 16:02:33.547331 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.547298    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219"} err="failed to get container status \"4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219\": rpc error: code = NotFound desc = could not find container \"4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219\": container with ID starting with 4e53c45c8fd9711cd90cd62b2217f57c951faa6c81e8a553ae727cd7f649b219 not found: ID does not exist"
Jun 02 16:02:33.565296 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.565264    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"]
Jun 02 16:02:33.581698 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:33.581673    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-86c5ff88dc-llpzc"]
Jun 02 16:02:35.433535 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:35.433503    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0e9efe9-51af-494f-ad97-d510698952e3" path="/var/lib/kubelet/pods/b0e9efe9-51af-494f-ad97-d510698952e3/volumes"
Jun 02 16:02:36.691952 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.691922    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"]
Jun 02 16:02:36.692321 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.692278    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e9efe9-51af-494f-ad97-d510698952e3" containerName="main"
Jun 02 16:02:36.692321 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.692290    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e9efe9-51af-494f-ad97-d510698952e3" containerName="main"
Jun 02 16:02:36.692321 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.692301    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0e9efe9-51af-494f-ad97-d510698952e3" containerName="storage-initializer"
Jun 02 16:02:36.692321 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.692307    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0e9efe9-51af-494f-ad97-d510698952e3" containerName="storage-initializer"
Jun 02 16:02:36.692450 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.692362    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0e9efe9-51af-494f-ad97-d510698952e3" containerName="main"
Jun 02 16:02:36.695912 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.695893    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.699328 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.699304    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 16:02:36.699479 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.699370    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8729g\""
Jun 02 16:02:36.700920 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.700898    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 16:02:36.701027 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.700963    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jun 02 16:02:36.709536 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.709512    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"]
Jun 02 16:02:36.822872 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.822835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-home\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.823042 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.822879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvk5g\" (UniqueName: \"kubernetes.io/projected/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kube-api-access-mvk5g\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.823042 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.822985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tls-certs\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.823132 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.823056    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-dshm\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.823132 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.823085    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.823132 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.823119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-model-cache\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.823268 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.823143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924307 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924274    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-dshm\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924307 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924312    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924526 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924334    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-model-cache\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924526 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924355    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924526 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924399    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-home\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924526 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924431    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mvk5g\" (UniqueName: \"kubernetes.io/projected/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kube-api-access-mvk5g\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924526 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924498    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tls-certs\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924742 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924721    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-model-cache\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924803 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924759    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924855 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924826    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tmp-dir\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.924946 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.924916    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-home\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.926755 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.926729    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-dshm\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.926944 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.926928    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tls-certs\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:36.933194 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:36.933152    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvk5g\" (UniqueName: \"kubernetes.io/projected/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kube-api-access-mvk5g\") pod \"precise-prefix-cache-test-kserve-6879b9c49c-h4shd\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:37.006657 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:37.006575    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:02:37.143742 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:37.143717    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"]
Jun 02 16:02:37.146314 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:02:37.146291    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbee5ab8_1158_4b23_ae0c_2a577956c8ea.slice/crio-59989a8a5ebcf844b50ad3eefca6b0e5203111e25ae614a490a4fe35c9a3dc4d WatchSource:0}: Error finding container 59989a8a5ebcf844b50ad3eefca6b0e5203111e25ae614a490a4fe35c9a3dc4d: Status 404 returned error can't find the container with id 59989a8a5ebcf844b50ad3eefca6b0e5203111e25ae614a490a4fe35c9a3dc4d
Jun 02 16:02:37.540549 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:37.540512    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd" event={"ID":"bbee5ab8-1158-4b23-ae0c-2a577956c8ea","Type":"ContainerStarted","Data":"7510c8a20554621b6db955936de0c124adcac5278cd3767bef9ae449f656bd36"}
Jun 02 16:02:37.540549 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:02:37.540551    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd" event={"ID":"bbee5ab8-1158-4b23-ae0c-2a577956c8ea","Type":"ContainerStarted","Data":"59989a8a5ebcf844b50ad3eefca6b0e5203111e25ae614a490a4fe35c9a3dc4d"}
Jun 02 16:03:13.656260 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:03:13.656232    2561 generic.go:358] "Generic (PLEG): container finished" podID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" containerID="7510c8a20554621b6db955936de0c124adcac5278cd3767bef9ae449f656bd36" exitCode=0
Jun 02 16:03:13.656600 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:03:13.656291    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd" event={"ID":"bbee5ab8-1158-4b23-ae0c-2a577956c8ea","Type":"ContainerDied","Data":"7510c8a20554621b6db955936de0c124adcac5278cd3767bef9ae449f656bd36"}
Jun 02 16:03:14.661022 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:03:14.660986    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd" event={"ID":"bbee5ab8-1158-4b23-ae0c-2a577956c8ea","Type":"ContainerStarted","Data":"09fa879dcfa7864a91a27d1b7d6e8787884b2cc03cd1f8fdf9e1534004ff324d"}
Jun 02 16:03:14.695136 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:03:14.695088    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd" podStartSLOduration=38.695074485 podStartE2EDuration="38.695074485s" podCreationTimestamp="2026-06-02 16:02:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:03:14.693965236 +0000 UTC m=+981.888744736" watchObservedRunningTime="2026-06-02 16:03:14.695074485 +0000 UTC m=+981.889853984"
Jun 02 16:03:17.006925 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:03:17.006888    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:03:17.006925 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:03:17.006925    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:03:17.019142 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:03:17.019115    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:03:17.685131 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:03:17.685100    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:06:53.394299 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:06:53.394273    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:06:53.396171 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:06:53.396131    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:06:53.400264 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:06:53.400243    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:06:53.402107 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:06:53.402087    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:11:53.418003 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:11:53.417976    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:11:53.421924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:11:53.421902    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:11:53.424012 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:11:53.423992    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:11:53.427272 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:11:53.427245    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:16:39.543521 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.543441    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"]
Jun 02 16:16:39.546112 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.546097    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.548973 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.548935    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jun 02 16:16:39.559873 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.559850    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"]
Jun 02 16:16:39.624291 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.624255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-model-cache\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.624516 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.624307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.624516 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.624391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-home\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.624516 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.624421    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-tmp-dir\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.624516 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.624444    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5180022-2ecb-47e4-b888-5c74c135d964-tls-certs\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.624516 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.624503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nx4r\" (UniqueName: \"kubernetes.io/projected/e5180022-2ecb-47e4-b888-5c74c135d964-kube-api-access-7nx4r\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.624765 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.624551    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-dshm\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725195 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725146    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7nx4r\" (UniqueName: \"kubernetes.io/projected/e5180022-2ecb-47e4-b888-5c74c135d964-kube-api-access-7nx4r\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725349 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725207    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-dshm\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725349 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725238    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-model-cache\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725349 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725267    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725349 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725301    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-home\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725526 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725439    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-tmp-dir\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725526 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725464    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5180022-2ecb-47e4-b888-5c74c135d964-tls-certs\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725740 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725714    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-model-cache\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725829 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725736    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725829 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725778    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-home\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.725902 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.725837    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-tmp-dir\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.727350 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.727330    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-dshm\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.727707 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.727677    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5180022-2ecb-47e4-b888-5c74c135d964-tls-certs\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.738323 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.738298    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nx4r\" (UniqueName: \"kubernetes.io/projected/e5180022-2ecb-47e4-b888-5c74c135d964-kube-api-access-7nx4r\") pod \"custom-route-timeout-test-kserve-77cf56cdbc-vwgm6\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.856710 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.856616    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:16:39.993654 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.993623    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"]
Jun 02 16:16:39.995489 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:16:39.995458    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5180022_2ecb_47e4_b888_5c74c135d964.slice/crio-7c39195d3c96c7c4b2c54669c993ca75d4fa3c024c9f7e2e8d96936aa455421f WatchSource:0}: Error finding container 7c39195d3c96c7c4b2c54669c993ca75d4fa3c024c9f7e2e8d96936aa455421f: Status 404 returned error can't find the container with id 7c39195d3c96c7c4b2c54669c993ca75d4fa3c024c9f7e2e8d96936aa455421f
Jun 02 16:16:39.997304 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:39.997288    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:16:40.329092 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:40.329033    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" event={"ID":"e5180022-2ecb-47e4-b888-5c74c135d964","Type":"ContainerStarted","Data":"4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202"}
Jun 02 16:16:40.329092 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:40.329093    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" event={"ID":"e5180022-2ecb-47e4-b888-5c74c135d964","Type":"ContainerStarted","Data":"7c39195d3c96c7c4b2c54669c993ca75d4fa3c024c9f7e2e8d96936aa455421f"}
Jun 02 16:16:44.344252 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:44.344149    2561 generic.go:358] "Generic (PLEG): container finished" podID="e5180022-2ecb-47e4-b888-5c74c135d964" containerID="4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202" exitCode=0
Jun 02 16:16:44.344252 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:44.344191    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" event={"ID":"e5180022-2ecb-47e4-b888-5c74c135d964","Type":"ContainerDied","Data":"4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202"}
Jun 02 16:16:53.443117 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:53.443017    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:16:53.487459 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:53.447002    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:16:53.487459 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:53.448510    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:16:53.487459 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:16:53.452457    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:17:33.527450 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:33.527417    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" event={"ID":"e5180022-2ecb-47e4-b888-5c74c135d964","Type":"ContainerStarted","Data":"5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655"}
Jun 02 16:17:33.552132 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:33.552083    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podStartSLOduration=6.264867752 podStartE2EDuration="54.552070977s" podCreationTimestamp="2026-06-02 16:16:39 +0000 UTC" firstStartedPulling="2026-06-02 16:16:44.34533733 +0000 UTC m=+1791.540116809" lastFinishedPulling="2026-06-02 16:17:32.632540552 +0000 UTC m=+1839.827320034" observedRunningTime="2026-06-02 16:17:33.550113905 +0000 UTC m=+1840.744893406" watchObservedRunningTime="2026-06-02 16:17:33.552070977 +0000 UTC m=+1840.746850486"
Jun 02 16:17:36.326574 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:36.326530    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"]
Jun 02 16:17:36.327038 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:36.326819    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd" podUID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" containerName="main" containerID="cri-o://09fa879dcfa7864a91a27d1b7d6e8787884b2cc03cd1f8fdf9e1534004ff324d" gracePeriod=30
Jun 02 16:17:36.538853 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:36.538812    2561 generic.go:358] "Generic (PLEG): container finished" podID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" containerID="09fa879dcfa7864a91a27d1b7d6e8787884b2cc03cd1f8fdf9e1534004ff324d" exitCode=0
Jun 02 16:17:36.539017 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:36.538872    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd" event={"ID":"bbee5ab8-1158-4b23-ae0c-2a577956c8ea","Type":"ContainerDied","Data":"09fa879dcfa7864a91a27d1b7d6e8787884b2cc03cd1f8fdf9e1534004ff324d"}
Jun 02 16:17:37.105842 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.105818    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:17:37.273082 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273006    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tls-certs\") pod \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") "
Jun 02 16:17:37.273082 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273058    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tmp-dir\") pod \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") "
Jun 02 16:17:37.273300 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273086    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvk5g\" (UniqueName: \"kubernetes.io/projected/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kube-api-access-mvk5g\") pod \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") "
Jun 02 16:17:37.273300 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273124    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-model-cache\") pod \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") "
Jun 02 16:17:37.273300 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273216    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-dshm\") pod \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") "
Jun 02 16:17:37.273300 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273270    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-home\") pod \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") "
Jun 02 16:17:37.273300 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273272    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bbee5ab8-1158-4b23-ae0c-2a577956c8ea" (UID: "bbee5ab8-1158-4b23-ae0c-2a577956c8ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:37.273300 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273296    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kserve-provision-location\") pod \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\" (UID: \"bbee5ab8-1158-4b23-ae0c-2a577956c8ea\") "
Jun 02 16:17:37.273596 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273473    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-home" (OuterVolumeSpecName: "home") pod "bbee5ab8-1158-4b23-ae0c-2a577956c8ea" (UID: "bbee5ab8-1158-4b23-ae0c-2a577956c8ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:37.273596 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273510    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "bbee5ab8-1158-4b23-ae0c-2a577956c8ea" (UID: "bbee5ab8-1158-4b23-ae0c-2a577956c8ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:37.273720 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273599    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:17:37.273720 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273641    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:17:37.273720 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.273657    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:17:37.275805 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.275775    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bbee5ab8-1158-4b23-ae0c-2a577956c8ea" (UID: "bbee5ab8-1158-4b23-ae0c-2a577956c8ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:17:37.275805 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.275787    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-dshm" (OuterVolumeSpecName: "dshm") pod "bbee5ab8-1158-4b23-ae0c-2a577956c8ea" (UID: "bbee5ab8-1158-4b23-ae0c-2a577956c8ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:37.275805 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.275800    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kube-api-access-mvk5g" (OuterVolumeSpecName: "kube-api-access-mvk5g") pod "bbee5ab8-1158-4b23-ae0c-2a577956c8ea" (UID: "bbee5ab8-1158-4b23-ae0c-2a577956c8ea"). InnerVolumeSpecName "kube-api-access-mvk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:17:37.328449 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.328402    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bbee5ab8-1158-4b23-ae0c-2a577956c8ea" (UID: "bbee5ab8-1158-4b23-ae0c-2a577956c8ea"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:17:37.374564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.374527    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:17:37.374564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.374556    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:17:37.374564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.374566    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:17:37.374564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.374576    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvk5g\" (UniqueName: \"kubernetes.io/projected/bbee5ab8-1158-4b23-ae0c-2a577956c8ea-kube-api-access-mvk5g\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:17:37.543692 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.543617    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"
Jun 02 16:17:37.543692 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.543636    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd" event={"ID":"bbee5ab8-1158-4b23-ae0c-2a577956c8ea","Type":"ContainerDied","Data":"59989a8a5ebcf844b50ad3eefca6b0e5203111e25ae614a490a4fe35c9a3dc4d"}
Jun 02 16:17:37.543692 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.543679    2561 scope.go:117] "RemoveContainer" containerID="09fa879dcfa7864a91a27d1b7d6e8787884b2cc03cd1f8fdf9e1534004ff324d"
Jun 02 16:17:37.551903 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.551885    2561 scope.go:117] "RemoveContainer" containerID="7510c8a20554621b6db955936de0c124adcac5278cd3767bef9ae449f656bd36"
Jun 02 16:17:37.565731 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.565707    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"]
Jun 02 16:17:37.570805 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:37.570781    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-6879b9c49c-h4shd"]
Jun 02 16:17:39.433470 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:39.433433    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" path="/var/lib/kubelet/pods/bbee5ab8-1158-4b23-ae0c-2a577956c8ea/volumes"
Jun 02 16:17:39.857025 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:39.856942    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:17:39.857025 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:39.857002    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:17:39.858704 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:39.858668    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:17:49.857623 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:49.857578    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:17:50.611260 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.611141    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"]
Jun 02 16:17:50.611694 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.611670    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" containerName="storage-initializer"
Jun 02 16:17:50.611694 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.611696    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" containerName="storage-initializer"
Jun 02 16:17:50.611872 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.611708    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" containerName="main"
Jun 02 16:17:50.611872 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.611714    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" containerName="main"
Jun 02 16:17:50.611872 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.611806    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="bbee5ab8-1158-4b23-ae0c-2a577956c8ea" containerName="main"
Jun 02 16:17:50.613932 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.613912    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.619856 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.619831    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jun 02 16:17:50.631033 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.631008    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"]
Jun 02 16:17:50.697319 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.697281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tmp-dir\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.697488 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.697334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzvw\" (UniqueName: \"kubernetes.io/projected/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kube-api-access-fmzvw\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.697488 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.697414    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-home\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.697488 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.697466    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tls-certs\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.697603 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.697507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-model-cache\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.697603 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.697536    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kserve-provision-location\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.697603 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.697591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-dshm\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798243 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798199    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-dshm\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798436 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798274    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tmp-dir\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798436 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798323    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzvw\" (UniqueName: \"kubernetes.io/projected/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kube-api-access-fmzvw\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798436 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798365    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-home\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798436 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798415    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tls-certs\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798656 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798448    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-model-cache\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798656 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798492    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kserve-provision-location\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798808 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798732    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tmp-dir\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798870 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798837    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-home\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.798964 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.798946    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-model-cache\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.799055 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.799038    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kserve-provision-location\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.800779 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.800757    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tls-certs\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.800937 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.800922    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-dshm\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.808330 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.808304    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzvw\" (UniqueName: \"kubernetes.io/projected/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kube-api-access-fmzvw\") pod \"stop-feature-test-kserve-6bcb45f7c5-w9k6m\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:50.926522 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:50.926484    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:17:51.080467 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:51.080440    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"]
Jun 02 16:17:51.082053 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:17:51.082021    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cebcdd_a3b7_4a5d_a88d_57a89cf7ef43.slice/crio-a076b26f3c38889becf0a97f35a627673c162e947a2876ac403d03042b09e71a WatchSource:0}: Error finding container a076b26f3c38889becf0a97f35a627673c162e947a2876ac403d03042b09e71a: Status 404 returned error can't find the container with id a076b26f3c38889becf0a97f35a627673c162e947a2876ac403d03042b09e71a
Jun 02 16:17:51.594564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:51.594524    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" event={"ID":"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43","Type":"ContainerStarted","Data":"47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482"}
Jun 02 16:17:51.594564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:51.594563    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" event={"ID":"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43","Type":"ContainerStarted","Data":"a076b26f3c38889becf0a97f35a627673c162e947a2876ac403d03042b09e71a"}
Jun 02 16:17:56.614301 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:56.614268    2561 generic.go:358] "Generic (PLEG): container finished" podID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerID="47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482" exitCode=0
Jun 02 16:17:56.614697 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:56.614339    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" event={"ID":"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43","Type":"ContainerDied","Data":"47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482"}
Jun 02 16:17:57.620348 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:57.620313    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" event={"ID":"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43","Type":"ContainerStarted","Data":"64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f"}
Jun 02 16:17:57.648773 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:57.648716    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podStartSLOduration=7.64869965 podStartE2EDuration="7.64869965s" podCreationTimestamp="2026-06-02 16:17:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:17:57.647147935 +0000 UTC m=+1864.841927436" watchObservedRunningTime="2026-06-02 16:17:57.64869965 +0000 UTC m=+1864.843479149"
Jun 02 16:17:59.857555 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:17:59.857512    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:18:00.927215 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:00.927173    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:18:00.927215 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:00.927223    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:18:00.928819 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:00.928785    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:18:09.857794 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:09.857678    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:18:10.927710 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:10.927657    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:18:19.857898 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:19.857846    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:18:20.927903 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:20.927863    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:18:29.858156 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:29.858103    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:18:30.927115 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:30.927062    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:18:39.857559 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:39.857502    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:18:40.927609 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:40.927555    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:18:49.858119 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:49.858071    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:18:50.927527 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:50.927472    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:18:59.857133 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:18:59.857084    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:19:00.927987 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:00.927943    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:19:09.857452 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:09.857396    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:19:10.927812 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:10.927757    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:19:19.857934 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:19.857889    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:19:20.927570 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:20.927523    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:19:29.858048 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:29.857997    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jun 02 16:19:30.927228 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:30.927177    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:19:39.867019 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:39.866935    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:19:39.879007 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:39.878977    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:19:40.927123 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:40.927079    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" probeResult="failure" output="Get \"https://10.132.0.34:8000/health\": dial tcp 10.132.0.34:8000: connect: connection refused"
Jun 02 16:19:49.483157 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:49.483113    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"]
Jun 02 16:19:49.483597 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:49.483507    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main" containerID="cri-o://5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655" gracePeriod=30
Jun 02 16:19:50.937070 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:50.937038    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:19:50.944983 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:19:50.944952    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:20:02.222143 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.222105    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"]
Jun 02 16:20:02.225691 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.225668    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.231259 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.231237    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jun 02 16:20:02.239264 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.239241    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"]
Jun 02 16:20:02.333357 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.333317    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kserve-provision-location\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.333543 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.333379    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-model-cache\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.333543 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.333434    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tmp-dir\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.333543 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.333457    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-home\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.333684 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.333541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-dshm\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.333684 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.333578    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tls-certs\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.333684 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.333638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrpgb\" (UniqueName: \"kubernetes.io/projected/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kube-api-access-wrpgb\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.434781 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.434736    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wrpgb\" (UniqueName: \"kubernetes.io/projected/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kube-api-access-wrpgb\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.434966 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.434793    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kserve-provision-location\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.434966 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.434825    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-model-cache\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.434966 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.434869    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tmp-dir\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.434966 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.434892    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-home\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.434966 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.434927    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-dshm\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.434966 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.434953    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tls-certs\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.435337 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.435311    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-model-cache\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.435454 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.435427    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-home\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.435454 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.435441    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tmp-dir\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.435598 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.435496    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kserve-provision-location\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.437231 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.437211    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-dshm\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.437411 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.437394    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tls-certs\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.443537 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.443511    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrpgb\" (UniqueName: \"kubernetes.io/projected/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kube-api-access-wrpgb\") pod \"router-with-refs-test-kserve-5ffc884955-9fpsh\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.535877 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.535786    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:20:02.674037 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:02.673992    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"]
Jun 02 16:20:02.676859 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:20:02.676834    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d8fe50_9eaa_4336_83bf_ec72125e07f8.slice/crio-31113bb3ae76dc2f571d1bdaf6e441e3808e2ab48186d01b3b5283adf791bef7 WatchSource:0}: Error finding container 31113bb3ae76dc2f571d1bdaf6e441e3808e2ab48186d01b3b5283adf791bef7: Status 404 returned error can't find the container with id 31113bb3ae76dc2f571d1bdaf6e441e3808e2ab48186d01b3b5283adf791bef7
Jun 02 16:20:03.086687 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:03.086649    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh" event={"ID":"e0d8fe50-9eaa-4336-83bf-ec72125e07f8","Type":"ContainerStarted","Data":"bdea0d3859a305de2db1e300ab8d8bf357ee3608d4391400b3c0181b38d0b669"}
Jun 02 16:20:03.086687 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:03.086688    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh" event={"ID":"e0d8fe50-9eaa-4336-83bf-ec72125e07f8","Type":"ContainerStarted","Data":"31113bb3ae76dc2f571d1bdaf6e441e3808e2ab48186d01b3b5283adf791bef7"}
Jun 02 16:20:03.770421 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:03.770383    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"]
Jun 02 16:20:03.770878 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:03.770742    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main" containerID="cri-o://64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f" gracePeriod=30
Jun 02 16:20:19.779719 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.779697    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-77cf56cdbc-vwgm6_e5180022-2ecb-47e4-b888-5c74c135d964/main/0.log"
Jun 02 16:20:19.780099 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.780080    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:20:19.911071 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911037    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-tmp-dir\") pod \"e5180022-2ecb-47e4-b888-5c74c135d964\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") "
Jun 02 16:20:19.911071 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911075    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nx4r\" (UniqueName: \"kubernetes.io/projected/e5180022-2ecb-47e4-b888-5c74c135d964-kube-api-access-7nx4r\") pod \"e5180022-2ecb-47e4-b888-5c74c135d964\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") "
Jun 02 16:20:19.911343 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911118    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-home\") pod \"e5180022-2ecb-47e4-b888-5c74c135d964\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") "
Jun 02 16:20:19.911343 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911181    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-model-cache\") pod \"e5180022-2ecb-47e4-b888-5c74c135d964\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") "
Jun 02 16:20:19.911343 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911208    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-dshm\") pod \"e5180022-2ecb-47e4-b888-5c74c135d964\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") "
Jun 02 16:20:19.911343 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911237    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5180022-2ecb-47e4-b888-5c74c135d964-tls-certs\") pod \"e5180022-2ecb-47e4-b888-5c74c135d964\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") "
Jun 02 16:20:19.911343 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911304    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-kserve-provision-location\") pod \"e5180022-2ecb-47e4-b888-5c74c135d964\" (UID: \"e5180022-2ecb-47e4-b888-5c74c135d964\") "
Jun 02 16:20:19.911595 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911481    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5180022-2ecb-47e4-b888-5c74c135d964" (UID: "e5180022-2ecb-47e4-b888-5c74c135d964"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:19.911650 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911601    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:19.911876 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.911848    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-home" (OuterVolumeSpecName: "home") pod "e5180022-2ecb-47e4-b888-5c74c135d964" (UID: "e5180022-2ecb-47e4-b888-5c74c135d964"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:19.913464 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.913427    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5180022-2ecb-47e4-b888-5c74c135d964-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5180022-2ecb-47e4-b888-5c74c135d964" (UID: "e5180022-2ecb-47e4-b888-5c74c135d964"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:20:19.913805 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.913776    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-dshm" (OuterVolumeSpecName: "dshm") pod "e5180022-2ecb-47e4-b888-5c74c135d964" (UID: "e5180022-2ecb-47e4-b888-5c74c135d964"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:19.913861 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.913789    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5180022-2ecb-47e4-b888-5c74c135d964-kube-api-access-7nx4r" (OuterVolumeSpecName: "kube-api-access-7nx4r") pod "e5180022-2ecb-47e4-b888-5c74c135d964" (UID: "e5180022-2ecb-47e4-b888-5c74c135d964"). InnerVolumeSpecName "kube-api-access-7nx4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:20:19.922125 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.922107    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5180022-2ecb-47e4-b888-5c74c135d964" (UID: "e5180022-2ecb-47e4-b888-5c74c135d964"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:19.973627 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:19.973581    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5180022-2ecb-47e4-b888-5c74c135d964" (UID: "e5180022-2ecb-47e4-b888-5c74c135d964"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:20.012440 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.012373    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:20.012440 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.012400    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5180022-2ecb-47e4-b888-5c74c135d964-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:20.012440 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.012411    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:20.012440 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.012421    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:20.012440 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.012430    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nx4r\" (UniqueName: \"kubernetes.io/projected/e5180022-2ecb-47e4-b888-5c74c135d964-kube-api-access-7nx4r\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:20.012440 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.012437    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5180022-2ecb-47e4-b888-5c74c135d964-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:20.146117 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.146093    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-test-kserve-77cf56cdbc-vwgm6_e5180022-2ecb-47e4-b888-5c74c135d964/main/0.log"
Jun 02 16:20:20.146475 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.146448    2561 generic.go:358] "Generic (PLEG): container finished" podID="e5180022-2ecb-47e4-b888-5c74c135d964" containerID="5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655" exitCode=137
Jun 02 16:20:20.146578 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.146510    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" event={"ID":"e5180022-2ecb-47e4-b888-5c74c135d964","Type":"ContainerDied","Data":"5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655"}
Jun 02 16:20:20.146578 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.146535    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"
Jun 02 16:20:20.146578 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.146547    2561 scope.go:117] "RemoveContainer" containerID="5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655"
Jun 02 16:20:20.146730 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.146538    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6" event={"ID":"e5180022-2ecb-47e4-b888-5c74c135d964","Type":"ContainerDied","Data":"7c39195d3c96c7c4b2c54669c993ca75d4fa3c024c9f7e2e8d96936aa455421f"}
Jun 02 16:20:20.156072 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.156032    2561 scope.go:117] "RemoveContainer" containerID="4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202"
Jun 02 16:20:20.168617 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.168600    2561 scope.go:117] "RemoveContainer" containerID="5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655"
Jun 02 16:20:20.168851 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:20:20.168829    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655\": container with ID starting with 5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655 not found: ID does not exist" containerID="5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655"
Jun 02 16:20:20.168920 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.168863    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655"} err="failed to get container status \"5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655\": rpc error: code = NotFound desc = could not find container \"5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655\": container with ID starting with 5865a3f53ed91f9762c73189033845d5693a818691ae4fb1820e492d4afbb655 not found: ID does not exist"
Jun 02 16:20:20.168920 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.168887    2561 scope.go:117] "RemoveContainer" containerID="4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202"
Jun 02 16:20:20.169116 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:20:20.169099    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202\": container with ID starting with 4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202 not found: ID does not exist" containerID="4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202"
Jun 02 16:20:20.169154 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.169121    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202"} err="failed to get container status \"4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202\": rpc error: code = NotFound desc = could not find container \"4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202\": container with ID starting with 4d878f79e75144bd530037d925070a69cb83f37f421546f623811cd74585e202 not found: ID does not exist"
Jun 02 16:20:20.173738 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.173718    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"]
Jun 02 16:20:20.179628 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:20.179606    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-77cf56cdbc-vwgm6"]
Jun 02 16:20:21.099883 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.099850    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"]
Jun 02 16:20:21.100339 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.100223    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main"
Jun 02 16:20:21.100339 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.100260    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main"
Jun 02 16:20:21.100339 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.100279    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="storage-initializer"
Jun 02 16:20:21.100339 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.100284    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="storage-initializer"
Jun 02 16:20:21.100544 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.100372    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" containerName="main"
Jun 02 16:20:21.139087 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.139045    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"]
Jun 02 16:20:21.139246 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.139216    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.225310 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.225270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-tls-certs\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.225463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.225329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-model-cache\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.225463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.225371    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-kserve-provision-location\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.225463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.225422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-dshm\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.225463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.225453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-home\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.225667 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.225468    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-tmp-dir\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.225667 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.225534    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vftch\" (UniqueName: \"kubernetes.io/projected/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-kube-api-access-vftch\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.326711 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.326671    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-model-cache\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.326711 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.326711    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-kserve-provision-location\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.326929 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.326756    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-dshm\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.326929 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.326784    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-home\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.326929 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.326803    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-tmp-dir\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.326929 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.326863    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vftch\" (UniqueName: \"kubernetes.io/projected/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-kube-api-access-vftch\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.326929 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.326903    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-tls-certs\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.327209 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.327156    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-model-cache\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.327305 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.327194    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-home\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.327305 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.327289    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-kserve-provision-location\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.327425 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.327384    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-tmp-dir\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.329228 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.329208    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-dshm\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.329474 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.329459    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-tls-certs\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.335849 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.335830    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vftch\" (UniqueName: \"kubernetes.io/projected/ffa2fa3f-ee94-4169-88d7-75768c2faa3a-kube-api-access-vftch\") pod \"stop-feature-test-kserve-6bcb45f7c5-2lt4v\" (UID: \"ffa2fa3f-ee94-4169-88d7-75768c2faa3a\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.433573 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.433538    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5180022-2ecb-47e4-b888-5c74c135d964" path="/var/lib/kubelet/pods/e5180022-2ecb-47e4-b888-5c74c135d964/volumes"
Jun 02 16:20:21.450947 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.450915    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"
Jun 02 16:20:21.589363 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:21.587439    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v"]
Jun 02 16:20:21.590205 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:20:21.590022    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa2fa3f_ee94_4169_88d7_75768c2faa3a.slice/crio-1890ecf68a115d967bfb00018e218442bc04c982fe7454ca344f3bc211b198a3 WatchSource:0}: Error finding container 1890ecf68a115d967bfb00018e218442bc04c982fe7454ca344f3bc211b198a3: Status 404 returned error can't find the container with id 1890ecf68a115d967bfb00018e218442bc04c982fe7454ca344f3bc211b198a3
Jun 02 16:20:22.157680 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:22.157642    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v" event={"ID":"ffa2fa3f-ee94-4169-88d7-75768c2faa3a","Type":"ContainerStarted","Data":"02c74651d54b98b160556987300db7ed958efb6f5eb444b720ea3ab065af0e6a"}
Jun 02 16:20:22.157680 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:22.157680    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-2lt4v" event={"ID":"ffa2fa3f-ee94-4169-88d7-75768c2faa3a","Type":"ContainerStarted","Data":"1890ecf68a115d967bfb00018e218442bc04c982fe7454ca344f3bc211b198a3"}
Jun 02 16:20:34.045795 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.045770    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-w9k6m_e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43/main/0.log"
Jun 02 16:20:34.046145 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.046111    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:20:34.145661 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.145633    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-dshm\") pod \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") "
Jun 02 16:20:34.145810 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.145686    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tls-certs\") pod \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") "
Jun 02 16:20:34.145810 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.145713    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kserve-provision-location\") pod \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") "
Jun 02 16:20:34.145810 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.145732    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tmp-dir\") pod \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") "
Jun 02 16:20:34.145810 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.145757    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-model-cache\") pod \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") "
Jun 02 16:20:34.145810 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.145782    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-home\") pod \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") "
Jun 02 16:20:34.146063 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.145813    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmzvw\" (UniqueName: \"kubernetes.io/projected/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kube-api-access-fmzvw\") pod \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\" (UID: \"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43\") "
Jun 02 16:20:34.146123 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.146048    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" (UID: "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:34.146608 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.146572    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-home" (OuterVolumeSpecName: "home") pod "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" (UID: "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:34.147944 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.147918    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" (UID: "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:20:34.148053 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.148011    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kube-api-access-fmzvw" (OuterVolumeSpecName: "kube-api-access-fmzvw") pod "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" (UID: "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43"). InnerVolumeSpecName "kube-api-access-fmzvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:20:34.148053 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.148019    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-dshm" (OuterVolumeSpecName: "dshm") pod "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" (UID: "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:34.158178 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.158142    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" (UID: "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:34.200906 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.200881    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-w9k6m_e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43/main/0.log"
Jun 02 16:20:34.201231 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.201207    2561 generic.go:358] "Generic (PLEG): container finished" podID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerID="64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f" exitCode=137
Jun 02 16:20:34.201338 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.201293    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"
Jun 02 16:20:34.201338 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.201295    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" event={"ID":"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43","Type":"ContainerDied","Data":"64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f"}
Jun 02 16:20:34.201445 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.201350    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m" event={"ID":"e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43","Type":"ContainerDied","Data":"a076b26f3c38889becf0a97f35a627673c162e947a2876ac403d03042b09e71a"}
Jun 02 16:20:34.201445 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.201371    2561 scope.go:117] "RemoveContainer" containerID="64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f"
Jun 02 16:20:34.201758 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.201727    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" (UID: "e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:20:34.210391 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.210374    2561 scope.go:117] "RemoveContainer" containerID="47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482"
Jun 02 16:20:34.247115 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.247087    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:34.247115 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.247114    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:34.247311 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.247124    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:34.247311 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.247134    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:34.247311 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.247145    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:34.247311 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.247153    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:34.247311 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.247178    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmzvw\" (UniqueName: \"kubernetes.io/projected/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43-kube-api-access-fmzvw\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:20:34.271844 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.271820    2561 scope.go:117] "RemoveContainer" containerID="64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f"
Jun 02 16:20:34.272198 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:20:34.272178    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f\": container with ID starting with 64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f not found: ID does not exist" containerID="64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f"
Jun 02 16:20:34.272259 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.272211    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f"} err="failed to get container status \"64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f\": rpc error: code = NotFound desc = could not find container \"64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f\": container with ID starting with 64e1825a824adccaf673b32e5ef2391b585961c4ed726cf21a7850218381842f not found: ID does not exist"
Jun 02 16:20:34.272259 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.272231    2561 scope.go:117] "RemoveContainer" containerID="47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482"
Jun 02 16:20:34.272501 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:20:34.272484    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482\": container with ID starting with 47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482 not found: ID does not exist" containerID="47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482"
Jun 02 16:20:34.272544 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.272512    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482"} err="failed to get container status \"47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482\": rpc error: code = NotFound desc = could not find container \"47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482\": container with ID starting with 47574a70f37d3aae027a3e5151f85fed0596e550c4eeb27a15767c17d3f46482 not found: ID does not exist"
Jun 02 16:20:34.526368 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.526335    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"]
Jun 02 16:20:34.529431 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:34.529406    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-6bcb45f7c5-w9k6m"]
Jun 02 16:20:35.433096 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:20:35.433057    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" path="/var/lib/kubelet/pods/e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43/volumes"
Jun 02 16:21:53.469818 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:21:53.469700    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:21:53.475240 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:21:53.475220    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:21:53.475785 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:21:53.475768    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:21:53.480537 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:21:53.480521    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:26:53.494307 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:26:53.494211    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:26:53.500228 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:26:53.500207    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:26:53.500640 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:26:53.500620    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:26:53.505976 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:26:53.505964    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:31:53.518679 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:31:53.518564    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:31:53.524207 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:31:53.524186    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:31:53.526110 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:31:53.526094    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:31:53.530983 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:31:53.530967    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:34:53.115466 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:53.115378    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"]
Jun 02 16:34:53.115961 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:53.115751    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh" podUID="e0d8fe50-9eaa-4336-83bf-ec72125e07f8" containerName="storage-initializer" containerID="cri-o://bdea0d3859a305de2db1e300ab8d8bf357ee3608d4391400b3c0181b38d0b669" gracePeriod=30
Jun 02 16:34:59.740717 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.740679    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"]
Jun 02 16:34:59.741296 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.741210    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="storage-initializer"
Jun 02 16:34:59.741296 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.741230    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="storage-initializer"
Jun 02 16:34:59.741296 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.741263    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main"
Jun 02 16:34:59.741296 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.741272    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main"
Jun 02 16:34:59.741517 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.741367    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9cebcdd-a3b7-4a5d-a88d-57a89cf7ef43" containerName="main"
Jun 02 16:34:59.743645 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.743623    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.746412 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.746390    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8f1a6f044e8c7a4d31a250e0c4861caf-kserve-self-signed-certs\""
Jun 02 16:34:59.746523 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.746391    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-dockercfg-tjqvn\""
Jun 02 16:34:59.754713 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.754689    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"]
Jun 02 16:34:59.782253 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.782224    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"]
Jun 02 16:34:59.784909 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.784888    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.795822 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.795798    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"]
Jun 02 16:34:59.829359 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829333    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqbmt\" (UniqueName: \"kubernetes.io/projected/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kube-api-access-kqbmt\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.829491 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.829491 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829403    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.829491 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829445    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lq4r\" (UniqueName: \"kubernetes.io/projected/acc30a5d-7287-4798-ba9d-c92972c26a21-kube-api-access-8lq4r\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.829491 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829462    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.829491 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829477    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.829655 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.829655 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829542    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.829655 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.829655 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829616    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.829655 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829642    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.829828 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829664    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.829828 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829679    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc30a5d-7287-4798-ba9d-c92972c26a21-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.829828 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.829699    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.930392 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930357    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.930561 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930405    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8lq4r\" (UniqueName: \"kubernetes.io/projected/acc30a5d-7287-4798-ba9d-c92972c26a21-kube-api-access-8lq4r\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.930561 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930434    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.930561 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930457    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.930561 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930485    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.930561 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930513    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.930861 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930577    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.930861 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930608    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.930861 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930794    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.930861 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930817    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.930861 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930840    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.931121 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930872    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.931121 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930904    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc30a5d-7287-4798-ba9d-c92972c26a21-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.931121 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930927    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.931121 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930940    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.931121 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.930981    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-home\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.931121 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.931026    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kqbmt\" (UniqueName: \"kubernetes.io/projected/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kube-api-access-kqbmt\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.931121 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.931061    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.931121 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.931111    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.931571 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.931179    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.931571 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.931202    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.931679 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.931577    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.932925 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.932898    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.933309 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.933286    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-dshm\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.933602 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.933579    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:34:59.933643 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.933626    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc30a5d-7287-4798-ba9d-c92972c26a21-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.939318 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.939291    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lq4r\" (UniqueName: \"kubernetes.io/projected/acc30a5d-7287-4798-ba9d-c92972c26a21-kube-api-access-8lq4r\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:34:59.940043 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:34:59.940026    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqbmt\" (UniqueName: \"kubernetes.io/projected/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kube-api-access-kqbmt\") pod \"llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:35:00.053330 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:00.053242    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:35:00.095551 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:00.095525    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:35:00.205252 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:00.205217    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"]
Jun 02 16:35:00.212457 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:00.212421    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:35:00.241238 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:00.241131    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"]
Jun 02 16:35:00.243428 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:35:00.243398    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc30a5d_7287_4798_ba9d_c92972c26a21.slice/crio-d3e8a6769375ce76d7b7bb252aeb3d17f33800dd9706b10c794ed530b2162689 WatchSource:0}: Error finding container d3e8a6769375ce76d7b7bb252aeb3d17f33800dd9706b10c794ed530b2162689: Status 404 returned error can't find the container with id d3e8a6769375ce76d7b7bb252aeb3d17f33800dd9706b10c794ed530b2162689
Jun 02 16:35:01.158808 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:01.158766    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" event={"ID":"cd74bf12-bcc2-4641-9d47-44467c1ca3bd","Type":"ContainerStarted","Data":"6515b813d2fb78d386a4620f68c7ba731b14d1e55c9d5e98232c0c71b614443e"}
Jun 02 16:35:01.160835 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:01.160775    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd" event={"ID":"acc30a5d-7287-4798-ba9d-c92972c26a21","Type":"ContainerStarted","Data":"7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f"}
Jun 02 16:35:01.160948 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:01.160846    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd" event={"ID":"acc30a5d-7287-4798-ba9d-c92972c26a21","Type":"ContainerStarted","Data":"d3e8a6769375ce76d7b7bb252aeb3d17f33800dd9706b10c794ed530b2162689"}
Jun 02 16:35:02.166068 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:02.166028    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" event={"ID":"cd74bf12-bcc2-4641-9d47-44467c1ca3bd","Type":"ContainerStarted","Data":"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1"}
Jun 02 16:35:02.166492 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:02.166128    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:35:03.172021 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:03.171985    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" event={"ID":"cd74bf12-bcc2-4641-9d47-44467c1ca3bd","Type":"ContainerStarted","Data":"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b"}
Jun 02 16:35:14.189138 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:14.189109    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:35:23.238919 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.238893    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-5ffc884955-9fpsh_e0d8fe50-9eaa-4336-83bf-ec72125e07f8/storage-initializer/0.log"
Jun 02 16:35:23.239307 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.238931    2561 generic.go:358] "Generic (PLEG): container finished" podID="e0d8fe50-9eaa-4336-83bf-ec72125e07f8" containerID="bdea0d3859a305de2db1e300ab8d8bf357ee3608d4391400b3c0181b38d0b669" exitCode=137
Jun 02 16:35:23.239307 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.239016    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh" event={"ID":"e0d8fe50-9eaa-4336-83bf-ec72125e07f8","Type":"ContainerDied","Data":"bdea0d3859a305de2db1e300ab8d8bf357ee3608d4391400b3c0181b38d0b669"}
Jun 02 16:35:23.309868 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.309846    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-5ffc884955-9fpsh_e0d8fe50-9eaa-4336-83bf-ec72125e07f8/storage-initializer/0.log"
Jun 02 16:35:23.309986 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.309913    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:35:23.450645 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.450617    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-dshm\") pod \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") "
Jun 02 16:35:23.450802 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.450658    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kserve-provision-location\") pod \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") "
Jun 02 16:35:23.450802 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.450675    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-model-cache\") pod \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") "
Jun 02 16:35:23.450802 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.450697    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tls-certs\") pod \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") "
Jun 02 16:35:23.450802 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.450724    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrpgb\" (UniqueName: \"kubernetes.io/projected/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kube-api-access-wrpgb\") pod \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") "
Jun 02 16:35:23.450802 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.450773    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tmp-dir\") pod \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") "
Jun 02 16:35:23.450802 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.450799    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-home\") pod \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\" (UID: \"e0d8fe50-9eaa-4336-83bf-ec72125e07f8\") "
Jun 02 16:35:23.451124 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.451012    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "e0d8fe50-9eaa-4336-83bf-ec72125e07f8" (UID: "e0d8fe50-9eaa-4336-83bf-ec72125e07f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:35:23.451124 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.451030    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e0d8fe50-9eaa-4336-83bf-ec72125e07f8" (UID: "e0d8fe50-9eaa-4336-83bf-ec72125e07f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:35:23.451265 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.451235    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:35:23.451265 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.451255    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:35:23.451265 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.451258    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-home" (OuterVolumeSpecName: "home") pod "e0d8fe50-9eaa-4336-83bf-ec72125e07f8" (UID: "e0d8fe50-9eaa-4336-83bf-ec72125e07f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:35:23.452821 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.452801    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-dshm" (OuterVolumeSpecName: "dshm") pod "e0d8fe50-9eaa-4336-83bf-ec72125e07f8" (UID: "e0d8fe50-9eaa-4336-83bf-ec72125e07f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:35:23.453259 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.453237    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e0d8fe50-9eaa-4336-83bf-ec72125e07f8" (UID: "e0d8fe50-9eaa-4336-83bf-ec72125e07f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:35:23.453259 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.453240    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kube-api-access-wrpgb" (OuterVolumeSpecName: "kube-api-access-wrpgb") pod "e0d8fe50-9eaa-4336-83bf-ec72125e07f8" (UID: "e0d8fe50-9eaa-4336-83bf-ec72125e07f8"). InnerVolumeSpecName "kube-api-access-wrpgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:35:23.504922 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.504892    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e0d8fe50-9eaa-4336-83bf-ec72125e07f8" (UID: "e0d8fe50-9eaa-4336-83bf-ec72125e07f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:35:23.551989 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.551964    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrpgb\" (UniqueName: \"kubernetes.io/projected/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kube-api-access-wrpgb\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:35:23.551989 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.551986    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:35:23.552216 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.551995    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:35:23.552216 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.552003    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:35:23.552216 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:23.552014    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0d8fe50-9eaa-4336-83bf-ec72125e07f8-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:35:24.243465 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:24.243441    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-test-kserve-5ffc884955-9fpsh_e0d8fe50-9eaa-4336-83bf-ec72125e07f8/storage-initializer/0.log"
Jun 02 16:35:24.243892 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:24.243573    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"
Jun 02 16:35:24.243892 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:24.243564    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh" event={"ID":"e0d8fe50-9eaa-4336-83bf-ec72125e07f8","Type":"ContainerDied","Data":"31113bb3ae76dc2f571d1bdaf6e441e3808e2ab48186d01b3b5283adf791bef7"}
Jun 02 16:35:24.243892 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:24.243690    2561 scope.go:117] "RemoveContainer" containerID="bdea0d3859a305de2db1e300ab8d8bf357ee3608d4391400b3c0181b38d0b669"
Jun 02 16:35:24.283677 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:24.283651    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"]
Jun 02 16:35:24.288847 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:24.288818    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-5ffc884955-9fpsh"]
Jun 02 16:35:25.432958 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:25.432926    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d8fe50-9eaa-4336-83bf-ec72125e07f8" path="/var/lib/kubelet/pods/e0d8fe50-9eaa-4336-83bf-ec72125e07f8/volumes"
Jun 02 16:35:58.368571 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:58.368534    2561 generic.go:358] "Generic (PLEG): container finished" podID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerID="50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b" exitCode=0
Jun 02 16:35:58.368970 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:58.368605    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" event={"ID":"cd74bf12-bcc2-4641-9d47-44467c1ca3bd","Type":"ContainerDied","Data":"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b"}
Jun 02 16:35:59.374979 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:59.374942    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" event={"ID":"cd74bf12-bcc2-4641-9d47-44467c1ca3bd","Type":"ContainerStarted","Data":"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3"}
Jun 02 16:35:59.404381 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:35:59.404314    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podStartSLOduration=58.875852703 podStartE2EDuration="1m0.404293372s" podCreationTimestamp="2026-06-02 16:34:59 +0000 UTC" firstStartedPulling="2026-06-02 16:35:00.212594569 +0000 UTC m=+2887.407374047" lastFinishedPulling="2026-06-02 16:35:01.741035235 +0000 UTC m=+2888.935814716" observedRunningTime="2026-06-02 16:35:59.401134609 +0000 UTC m=+2946.595914108" watchObservedRunningTime="2026-06-02 16:35:59.404293372 +0000 UTC m=+2946.599072874"
Jun 02 16:36:00.054298 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:00.054261    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:36:00.054298 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:00.054305    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:36:00.054674 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:00.054642    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:36:10.054114 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:10.054008    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:36:20.054372 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:20.054324    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:36:24.513114 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.513079    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"]
Jun 02 16:36:24.513494 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.513453    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d8fe50-9eaa-4336-83bf-ec72125e07f8" containerName="storage-initializer"
Jun 02 16:36:24.513494 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.513465    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d8fe50-9eaa-4336-83bf-ec72125e07f8" containerName="storage-initializer"
Jun 02 16:36:24.513569 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.513542    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e0d8fe50-9eaa-4336-83bf-ec72125e07f8" containerName="storage-initializer"
Jun 02 16:36:24.516414 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.516398    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.519299 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.519276    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8de1d74aab16d9cabd8b5aafeb5248e8-kserve-self-signed-certs\""
Jun 02 16:36:24.527131 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.527106    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"]
Jun 02 16:36:24.622073 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.622039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.622258 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.622086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jdp7\" (UniqueName: \"kubernetes.io/projected/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kube-api-access-8jdp7\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.622258 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.622112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.622258 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.622220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.622386 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.622263    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.622386 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.622283    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.622386 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.622356    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.722885 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.722839    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.722885 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.722890    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723090 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723033    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdp7\" (UniqueName: \"kubernetes.io/projected/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kube-api-access-8jdp7\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723133 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723094    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723198    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723237    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723258    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723279    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723286    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723503    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.723530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.723501    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-home\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.725503 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.725474    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.725980 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.725960    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.731366 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.731340    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdp7\" (UniqueName: \"kubernetes.io/projected/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kube-api-access-8jdp7\") pod \"llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.827994 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.827897    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:36:24.971892 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:24.971868    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"]
Jun 02 16:36:24.974367 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:36:24.974340    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69bad32_fe1a_4556_8718_f8f3937d7dc1.slice/crio-84e351838cdd77ea3f5a6635351612952247eb26d541b800cb4b3634fbdbb544 WatchSource:0}: Error finding container 84e351838cdd77ea3f5a6635351612952247eb26d541b800cb4b3634fbdbb544: Status 404 returned error can't find the container with id 84e351838cdd77ea3f5a6635351612952247eb26d541b800cb4b3634fbdbb544
Jun 02 16:36:25.470122 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:25.470086    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4" event={"ID":"a69bad32-fe1a-4556-8718-f8f3937d7dc1","Type":"ContainerStarted","Data":"acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655"}
Jun 02 16:36:25.470382 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:25.470130    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4" event={"ID":"a69bad32-fe1a-4556-8718-f8f3937d7dc1","Type":"ContainerStarted","Data":"84e351838cdd77ea3f5a6635351612952247eb26d541b800cb4b3634fbdbb544"}
Jun 02 16:36:30.054463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:30.054408    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:36:40.053868 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:40.053828    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:36:50.053944 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:50.053891    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:36:53.546796 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:53.546674    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:36:53.552719 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:53.552698    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:36:53.553613 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:53.553593    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:36:53.559686 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:36:53.559669    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:37:00.054331 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:37:00.054276    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:37:10.053689 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:37:10.053644    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:37:20.054199 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:37:20.054131    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.37:8001/health\": dial tcp 10.132.0.37:8001: connect: connection refused"
Jun 02 16:37:30.066706 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:37:30.066677    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:37:30.078623 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:37:30.078600    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:41:53.573444 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:41:53.573327    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:41:53.579572 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:41:53.579549    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:41:53.581530 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:41:53.581513    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:41:53.586942 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:41:53.586926    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:46:53.602946 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:46:53.602835    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:46:53.608570 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:46:53.608548    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:46:53.609366 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:46:53.609345    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:46:53.614789 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:46:53.614772    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:49:55.021273 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:49:55.021185    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"]
Jun 02 16:49:55.021939 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:49:55.021648    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main" containerID="cri-o://348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3" gracePeriod=30
Jun 02 16:49:55.025426 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:49:55.025401    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"]
Jun 02 16:49:55.025863 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:49:55.025817    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd" podUID="acc30a5d-7287-4798-ba9d-c92972c26a21" containerName="storage-initializer" containerID="cri-o://7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f" gracePeriod=30
Jun 02 16:50:02.340337 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.340302    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"]
Jun 02 16:50:02.344194 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.344151    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.346801 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.346779    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-9qrtd\""
Jun 02 16:50:02.346995 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.346982    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jun 02 16:50:02.351680 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.351463    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"]
Jun 02 16:50:02.357729 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.357708    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"]
Jun 02 16:50:02.361434 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.361410    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.375529 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.375501    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"]
Jun 02 16:50:02.396461 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.396581 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396470    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.396581 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.396581 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396566    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.396692 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.396692 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396627    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84105eba-2b85-400c-8627-7f6d6ce0157a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.396692 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396657    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-home\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.396692 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396686    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.396821 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.396821 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396740    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.396821 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396758    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.396821 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhrwz\" (UniqueName: \"kubernetes.io/projected/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kube-api-access-hhrwz\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.396946 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-dshm\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.396946 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.396856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn87s\" (UniqueName: \"kubernetes.io/projected/84105eba-2b85-400c-8627-7f6d6ce0157a-kube-api-access-tn87s\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.498308 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498279    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.498463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498326    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.498463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498352    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84105eba-2b85-400c-8627-7f6d6ce0157a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.498463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498371    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-home\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.498463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498394    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.498463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498430    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.498463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498450    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.498787 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498478    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.498787 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498530    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hhrwz\" (UniqueName: \"kubernetes.io/projected/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kube-api-access-hhrwz\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.498787 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498574    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-dshm\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.498787 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498618    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tn87s\" (UniqueName: \"kubernetes.io/projected/84105eba-2b85-400c-8627-7f6d6ce0157a-kube-api-access-tn87s\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.498787 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498664    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.498787 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498696    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.498787 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498731    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.498787 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498736    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.499244 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.498906    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-model-cache\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.499244 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.499013    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-home\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.499244 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.499108    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.499244 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.499194    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.499575 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.499528    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.499697 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.499623    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.499771 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.499730    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.501149 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.501128    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.501279 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.501258    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.501346 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.501292    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84105eba-2b85-400c-8627-7f6d6ce0157a-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.501419 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.501398    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-dshm\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.506901 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.506874    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhrwz\" (UniqueName: \"kubernetes.io/projected/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kube-api-access-hhrwz\") pod \"custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.506981 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.506972    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn87s\" (UniqueName: \"kubernetes.io/projected/84105eba-2b85-400c-8627-7f6d6ce0157a-kube-api-access-tn87s\") pod \"custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.654645 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.654615    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:02.674560 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.674528    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:02.800880 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.800853    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"]
Jun 02 16:50:02.802967 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:50:02.802926    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d28bcf_1993_4bd0_87be_2215aa3970b0.slice/crio-bf4fcc5cb5ab9bdf6e881364b62a4fcbc92dc6d917e93b8093449bf8e0db4874 WatchSource:0}: Error finding container bf4fcc5cb5ab9bdf6e881364b62a4fcbc92dc6d917e93b8093449bf8e0db4874: Status 404 returned error can't find the container with id bf4fcc5cb5ab9bdf6e881364b62a4fcbc92dc6d917e93b8093449bf8e0db4874
Jun 02 16:50:02.805029 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.805008    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:50:02.816951 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:02.816933    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"]
Jun 02 16:50:02.818998 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:50:02.818974    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84105eba_2b85_400c_8627_7f6d6ce0157a.slice/crio-ac94c97252209271084b865041ab1efdbc09075bcd4969cc04fd0cd71d0ca5d2 WatchSource:0}: Error finding container ac94c97252209271084b865041ab1efdbc09075bcd4969cc04fd0cd71d0ca5d2: Status 404 returned error can't find the container with id ac94c97252209271084b865041ab1efdbc09075bcd4969cc04fd0cd71d0ca5d2
Jun 02 16:50:03.325521 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:03.325482    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" event={"ID":"84105eba-2b85-400c-8627-7f6d6ce0157a","Type":"ContainerStarted","Data":"187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b"}
Jun 02 16:50:03.325521 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:03.325520    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" event={"ID":"84105eba-2b85-400c-8627-7f6d6ce0157a","Type":"ContainerStarted","Data":"ac94c97252209271084b865041ab1efdbc09075bcd4969cc04fd0cd71d0ca5d2"}
Jun 02 16:50:03.326932 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:03.326903    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" event={"ID":"e2d28bcf-1993-4bd0-87be-2215aa3970b0","Type":"ContainerStarted","Data":"7745e9a205e76848e8bca75d92a5dc3c1c52bfaaf0b5b39b47c88afbaf4230c5"}
Jun 02 16:50:03.327065 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:03.326936    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" event={"ID":"e2d28bcf-1993-4bd0-87be-2215aa3970b0","Type":"ContainerStarted","Data":"bf4fcc5cb5ab9bdf6e881364b62a4fcbc92dc6d917e93b8093449bf8e0db4874"}
Jun 02 16:50:03.327065 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:03.327050    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:04.333468 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:04.333429    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" event={"ID":"e2d28bcf-1993-4bd0-87be-2215aa3970b0","Type":"ContainerStarted","Data":"5679a92aab0602f1ef8accb08d3d39d4a9c914b80d671bcfc9455609572eaabb"}
Jun 02 16:50:07.347191 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:07.347088    2561 generic.go:358] "Generic (PLEG): container finished" podID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerID="187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b" exitCode=0
Jun 02 16:50:07.347191 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:07.347172    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" event={"ID":"84105eba-2b85-400c-8627-7f6d6ce0157a","Type":"ContainerDied","Data":"187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b"}
Jun 02 16:50:08.353029 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:08.352990    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" event={"ID":"84105eba-2b85-400c-8627-7f6d6ce0157a","Type":"ContainerStarted","Data":"f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda"}
Jun 02 16:50:08.376741 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:08.376687    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podStartSLOduration=6.376672858 podStartE2EDuration="6.376672858s" podCreationTimestamp="2026-06-02 16:50:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:50:08.375150004 +0000 UTC m=+3795.569929515" watchObservedRunningTime="2026-06-02 16:50:08.376672858 +0000 UTC m=+3795.571452358"
Jun 02 16:50:12.675565 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:12.675535    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:12.676019 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:12.675580    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:50:12.676774 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:12.676748    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:50:15.349863 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:15.349833    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 16:50:22.675777 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:22.675729    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:50:25.021940 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.021876    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="llm-d-routing-sidecar" containerID="cri-o://732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1" gracePeriod=2
Jun 02 16:50:25.304393 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.304368    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd_acc30a5d-7287-4798-ba9d-c92972c26a21/storage-initializer/0.log"
Jun 02 16:50:25.304520 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.304442    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:50:25.330586 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.330559    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm_cd74bf12-bcc2-4641-9d47-44467c1ca3bd/main/0.log"
Jun 02 16:50:25.331307 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.331289    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:50:25.422102 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.422077    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm_cd74bf12-bcc2-4641-9d47-44467c1ca3bd/main/0.log"
Jun 02 16:50:25.422665 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.422638    2561 generic.go:358] "Generic (PLEG): container finished" podID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerID="348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3" exitCode=137
Jun 02 16:50:25.422665 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.422664    2561 generic.go:358] "Generic (PLEG): container finished" podID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerID="732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1" exitCode=0
Jun 02 16:50:25.422837 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.422670    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" event={"ID":"cd74bf12-bcc2-4641-9d47-44467c1ca3bd","Type":"ContainerDied","Data":"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3"}
Jun 02 16:50:25.422837 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.422711    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" event={"ID":"cd74bf12-bcc2-4641-9d47-44467c1ca3bd","Type":"ContainerDied","Data":"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1"}
Jun 02 16:50:25.422837 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.422723    2561 scope.go:117] "RemoveContainer" containerID="348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3"
Jun 02 16:50:25.422837 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.422724    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm" event={"ID":"cd74bf12-bcc2-4641-9d47-44467c1ca3bd","Type":"ContainerDied","Data":"6515b813d2fb78d386a4620f68c7ba731b14d1e55c9d5e98232c0c71b614443e"}
Jun 02 16:50:25.422837 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.422711    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"
Jun 02 16:50:25.423959 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.423943    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd_acc30a5d-7287-4798-ba9d-c92972c26a21/storage-initializer/0.log"
Jun 02 16:50:25.424068 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.423975    2561 generic.go:358] "Generic (PLEG): container finished" podID="acc30a5d-7287-4798-ba9d-c92972c26a21" containerID="7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f" exitCode=137
Jun 02 16:50:25.424068 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.424013    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd" event={"ID":"acc30a5d-7287-4798-ba9d-c92972c26a21","Type":"ContainerDied","Data":"7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f"}
Jun 02 16:50:25.424068 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.424031    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd" event={"ID":"acc30a5d-7287-4798-ba9d-c92972c26a21","Type":"ContainerDied","Data":"d3e8a6769375ce76d7b7bb252aeb3d17f33800dd9706b10c794ed530b2162689"}
Jun 02 16:50:25.424068 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.424052    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"
Jun 02 16:50:25.430276 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430255    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lq4r\" (UniqueName: \"kubernetes.io/projected/acc30a5d-7287-4798-ba9d-c92972c26a21-kube-api-access-8lq4r\") pod \"acc30a5d-7287-4798-ba9d-c92972c26a21\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") "
Jun 02 16:50:25.430377 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430286    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-home\") pod \"acc30a5d-7287-4798-ba9d-c92972c26a21\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") "
Jun 02 16:50:25.430377 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430317    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-home\") pod \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") "
Jun 02 16:50:25.430377 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430340    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kserve-provision-location\") pod \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430382    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tmp-dir\") pod \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430421    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-tmp-dir\") pod \"acc30a5d-7287-4798-ba9d-c92972c26a21\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430466    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-model-cache\") pod \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430574    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-home" (OuterVolumeSpecName: "home") pod "acc30a5d-7287-4798-ba9d-c92972c26a21" (UID: "acc30a5d-7287-4798-ba9d-c92972c26a21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430596    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc30a5d-7287-4798-ba9d-c92972c26a21-tls-certs\") pod \"acc30a5d-7287-4798-ba9d-c92972c26a21\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430626    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-dshm\") pod \"acc30a5d-7287-4798-ba9d-c92972c26a21\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430731    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-dshm\") pod \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430764    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-kserve-provision-location\") pod \"acc30a5d-7287-4798-ba9d-c92972c26a21\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430776    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acc30a5d-7287-4798-ba9d-c92972c26a21" (UID: "acc30a5d-7287-4798-ba9d-c92972c26a21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430795    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kqbmt\" (UniqueName: \"kubernetes.io/projected/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kube-api-access-kqbmt\") pod \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430832    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-model-cache\") pod \"acc30a5d-7287-4798-ba9d-c92972c26a21\" (UID: \"acc30a5d-7287-4798-ba9d-c92972c26a21\") "
Jun 02 16:50:25.430882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.430871    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tls-certs\") pod \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\" (UID: \"cd74bf12-bcc2-4641-9d47-44467c1ca3bd\") "
Jun 02 16:50:25.431546 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.431023    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-home" (OuterVolumeSpecName: "home") pod "cd74bf12-bcc2-4641-9d47-44467c1ca3bd" (UID: "cd74bf12-bcc2-4641-9d47-44467c1ca3bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.431546 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.431237    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.431546 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.431256    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.431546 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.431270    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.433841 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.432735    2561 scope.go:117] "RemoveContainer" containerID="50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b"
Jun 02 16:50:25.433841 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.432888    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc30a5d-7287-4798-ba9d-c92972c26a21-kube-api-access-8lq4r" (OuterVolumeSpecName: "kube-api-access-8lq4r") pod "acc30a5d-7287-4798-ba9d-c92972c26a21" (UID: "acc30a5d-7287-4798-ba9d-c92972c26a21"). InnerVolumeSpecName "kube-api-access-8lq4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:50:25.433988 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.433945    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-model-cache" (OuterVolumeSpecName: "model-cache") pod "acc30a5d-7287-4798-ba9d-c92972c26a21" (UID: "acc30a5d-7287-4798-ba9d-c92972c26a21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.434344 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.434127    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd74bf12-bcc2-4641-9d47-44467c1ca3bd" (UID: "cd74bf12-bcc2-4641-9d47-44467c1ca3bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.435310 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.435284    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc30a5d-7287-4798-ba9d-c92972c26a21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acc30a5d-7287-4798-ba9d-c92972c26a21" (UID: "acc30a5d-7287-4798-ba9d-c92972c26a21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:50:25.435310 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.435313    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-dshm" (OuterVolumeSpecName: "dshm") pod "cd74bf12-bcc2-4641-9d47-44467c1ca3bd" (UID: "cd74bf12-bcc2-4641-9d47-44467c1ca3bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.435472 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.435430    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd74bf12-bcc2-4641-9d47-44467c1ca3bd" (UID: "cd74bf12-bcc2-4641-9d47-44467c1ca3bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:50:25.436188 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.436092    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-dshm" (OuterVolumeSpecName: "dshm") pod "acc30a5d-7287-4798-ba9d-c92972c26a21" (UID: "acc30a5d-7287-4798-ba9d-c92972c26a21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.437477 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.437444    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kube-api-access-kqbmt" (OuterVolumeSpecName: "kube-api-access-kqbmt") pod "cd74bf12-bcc2-4641-9d47-44467c1ca3bd" (UID: "cd74bf12-bcc2-4641-9d47-44467c1ca3bd"). InnerVolumeSpecName "kube-api-access-kqbmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:50:25.449861 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.449837    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd74bf12-bcc2-4641-9d47-44467c1ca3bd" (UID: "cd74bf12-bcc2-4641-9d47-44467c1ca3bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.475816 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.475784    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "acc30a5d-7287-4798-ba9d-c92972c26a21" (UID: "acc30a5d-7287-4798-ba9d-c92972c26a21"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.500731 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.500691    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd74bf12-bcc2-4641-9d47-44467c1ca3bd" (UID: "cd74bf12-bcc2-4641-9d47-44467c1ca3bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:50:25.508257 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.508235    2561 scope.go:117] "RemoveContainer" containerID="732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1"
Jun 02 16:50:25.517879 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.517859    2561 scope.go:117] "RemoveContainer" containerID="348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3"
Jun 02 16:50:25.518185 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:50:25.518139    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3\": container with ID starting with 348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3 not found: ID does not exist" containerID="348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3"
Jun 02 16:50:25.518298 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.518193    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3"} err="failed to get container status \"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3\": rpc error: code = NotFound desc = could not find container \"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3\": container with ID starting with 348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3 not found: ID does not exist"
Jun 02 16:50:25.518298 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.518213    2561 scope.go:117] "RemoveContainer" containerID="50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b"
Jun 02 16:50:25.518444 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:50:25.518425    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b\": container with ID starting with 50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b not found: ID does not exist" containerID="50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b"
Jun 02 16:50:25.518505 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.518449    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b"} err="failed to get container status \"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b\": rpc error: code = NotFound desc = could not find container \"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b\": container with ID starting with 50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b not found: ID does not exist"
Jun 02 16:50:25.518505 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.518466    2561 scope.go:117] "RemoveContainer" containerID="732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1"
Jun 02 16:50:25.518717 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:50:25.518696    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1\": container with ID starting with 732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1 not found: ID does not exist" containerID="732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1"
Jun 02 16:50:25.518786 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.518719    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1"} err="failed to get container status \"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1\": rpc error: code = NotFound desc = could not find container \"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1\": container with ID starting with 732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1 not found: ID does not exist"
Jun 02 16:50:25.518786 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.518735    2561 scope.go:117] "RemoveContainer" containerID="348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3"
Jun 02 16:50:25.518951 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.518930    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3"} err="failed to get container status \"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3\": rpc error: code = NotFound desc = could not find container \"348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3\": container with ID starting with 348c8026c4725823293b7b771280ee375e8e31640e38b7a5e104a23193d26ce3 not found: ID does not exist"
Jun 02 16:50:25.518951 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.518948    2561 scope.go:117] "RemoveContainer" containerID="50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b"
Jun 02 16:50:25.519134 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.519116    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b"} err="failed to get container status \"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b\": rpc error: code = NotFound desc = could not find container \"50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b\": container with ID starting with 50723962d7d6038035f2e6f5bb04efff2c833d5657d8193ca1f962821a12ac8b not found: ID does not exist"
Jun 02 16:50:25.519134 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.519132    2561 scope.go:117] "RemoveContainer" containerID="732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1"
Jun 02 16:50:25.519343 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.519325    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1"} err="failed to get container status \"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1\": rpc error: code = NotFound desc = could not find container \"732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1\": container with ID starting with 732fab5a1d4033d4305f6c099aafddcf8c6c838489ec16d4e8992728ef16efa1 not found: ID does not exist"
Jun 02 16:50:25.519411 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.519345    2561 scope.go:117] "RemoveContainer" containerID="7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f"
Jun 02 16:50:25.532798 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532767    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acc30a5d-7287-4798-ba9d-c92972c26a21-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532798 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532796    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532809    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532824    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532841    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqbmt\" (UniqueName: \"kubernetes.io/projected/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kube-api-access-kqbmt\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532856    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acc30a5d-7287-4798-ba9d-c92972c26a21-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532870    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532884    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lq4r\" (UniqueName: \"kubernetes.io/projected/acc30a5d-7287-4798-ba9d-c92972c26a21-kube-api-access-8lq4r\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532898    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.532924 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532912    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.533174 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.532927    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd74bf12-bcc2-4641-9d47-44467c1ca3bd-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:50:25.567931 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.567908    2561 scope.go:117] "RemoveContainer" containerID="7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f"
Jun 02 16:50:25.568273 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:50:25.568252    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f\": container with ID starting with 7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f not found: ID does not exist" containerID="7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f"
Jun 02 16:50:25.568348 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.568281    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f"} err="failed to get container status \"7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f\": rpc error: code = NotFound desc = could not find container \"7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f\": container with ID starting with 7188c94077c3aeb07723185fdfc3b9448d36ece07b97013f54ccc8ba212e6a4f not found: ID does not exist"
Jun 02 16:50:25.748132 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.748107    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"]
Jun 02 16:50:25.752461 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.752434    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-65958d85cdnngsm"]
Jun 02 16:50:25.779457 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.779428    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"]
Jun 02 16:50:25.786207 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:25.786176    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-05aa9bba-kserve-prefill-6d4b9dd"]
Jun 02 16:50:27.433206 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:27.433154    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acc30a5d-7287-4798-ba9d-c92972c26a21" path="/var/lib/kubelet/pods/acc30a5d-7287-4798-ba9d-c92972c26a21/volumes"
Jun 02 16:50:27.433598 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:27.433584    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" path="/var/lib/kubelet/pods/cd74bf12-bcc2-4641-9d47-44467c1ca3bd/volumes"
Jun 02 16:50:32.675189 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:32.675134    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:50:42.675301 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:42.675263    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:50:52.675851 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:50:52.675796    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:51:02.675249 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:02.675213    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:51:12.675489 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:12.675401    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:51:15.842465 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:15.842432    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"]
Jun 02 16:51:15.842875 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:15.842778    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4" podUID="a69bad32-fe1a-4556-8718-f8f3937d7dc1" containerName="storage-initializer" containerID="cri-o://acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655" gracePeriod=30
Jun 02 16:51:22.675825 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:22.675784    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:51:24.098749 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.098718    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jun 02 16:51:24.099190 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099117    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main"
Jun 02 16:51:24.099190 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099129    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main"
Jun 02 16:51:24.099190 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099145    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="storage-initializer"
Jun 02 16:51:24.099190 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099153    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="storage-initializer"
Jun 02 16:51:24.099190 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099184    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acc30a5d-7287-4798-ba9d-c92972c26a21" containerName="storage-initializer"
Jun 02 16:51:24.099190 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099194    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="acc30a5d-7287-4798-ba9d-c92972c26a21" containerName="storage-initializer"
Jun 02 16:51:24.099416 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099210    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="llm-d-routing-sidecar"
Jun 02 16:51:24.099416 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099216    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="llm-d-routing-sidecar"
Jun 02 16:51:24.099416 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099281    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="acc30a5d-7287-4798-ba9d-c92972c26a21" containerName="storage-initializer"
Jun 02 16:51:24.099416 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099291    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="main"
Jun 02 16:51:24.099416 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.099298    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="cd74bf12-bcc2-4641-9d47-44467c1ca3bd" containerName="llm-d-routing-sidecar"
Jun 02 16:51:24.102014 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.101985    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.104629 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.104608    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-pf4zd\""
Jun 02 16:51:24.105603 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.105585    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\""
Jun 02 16:51:24.114490 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.114465    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jun 02 16:51:24.186979 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.186945    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.187236 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.187025    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.187236 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.187057    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvvqc\" (UniqueName: \"kubernetes.io/projected/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kube-api-access-fvvqc\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.187236 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.187129    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.187375 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.187228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.187375 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.187273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.187375 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.187331    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288372 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288334    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288386    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288408    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288441    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288465    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288513    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288564 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288543    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fvvqc\" (UniqueName: \"kubernetes.io/projected/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kube-api-access-fvvqc\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288882 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288819    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288934 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288894    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.288977 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.288922    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.289031 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.289015    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.290599 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.290580    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.291060 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.291047    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.296027 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.295999    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvvqc\" (UniqueName: \"kubernetes.io/projected/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kube-api-access-fvvqc\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.413345 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.413308    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:51:24.551911 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.551876    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jun 02 16:51:24.556535 ip-10-0-135-231 kubenswrapper[2561]: W0602 16:51:24.556478    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1dc8002_9308_4bac_a2a5_477df9e2cce7.slice/crio-ee3be610bdd2483eee7dcc37745038cae3227fdf556e4baf82a1734c159dd404 WatchSource:0}: Error finding container ee3be610bdd2483eee7dcc37745038cae3227fdf556e4baf82a1734c159dd404: Status 404 returned error can't find the container with id ee3be610bdd2483eee7dcc37745038cae3227fdf556e4baf82a1734c159dd404
Jun 02 16:51:24.632745 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.632715    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1dc8002-9308-4bac-a2a5-477df9e2cce7","Type":"ContainerStarted","Data":"e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c"}
Jun 02 16:51:24.632847 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:24.632755    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1dc8002-9308-4bac-a2a5-477df9e2cce7","Type":"ContainerStarted","Data":"ee3be610bdd2483eee7dcc37745038cae3227fdf556e4baf82a1734c159dd404"}
Jun 02 16:51:32.675905 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:32.675862    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.41:8000/health\": dial tcp 10.132.0.41:8000: connect: connection refused"
Jun 02 16:51:42.685279 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:42.685252    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:51:42.692888 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:42.692857    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 16:51:46.031308 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.031281    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4_a69bad32-fe1a-4556-8718-f8f3937d7dc1/storage-initializer/0.log"
Jun 02 16:51:46.031601 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.031353    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:51:46.205700 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.205669    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jdp7\" (UniqueName: \"kubernetes.io/projected/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kube-api-access-8jdp7\") pod \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") "
Jun 02 16:51:46.205862 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.205707    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-dshm\") pod \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") "
Jun 02 16:51:46.205862 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.205732    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kserve-provision-location\") pod \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") "
Jun 02 16:51:46.205862 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.205770    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-home\") pod \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") "
Jun 02 16:51:46.205862 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.205808    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tmp-dir\") pod \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") "
Jun 02 16:51:46.205862 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.205833    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-model-cache\") pod \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") "
Jun 02 16:51:46.205862 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.205861    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tls-certs\") pod \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\" (UID: \"a69bad32-fe1a-4556-8718-f8f3937d7dc1\") "
Jun 02 16:51:46.206201 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.206100    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a69bad32-fe1a-4556-8718-f8f3937d7dc1" (UID: "a69bad32-fe1a-4556-8718-f8f3937d7dc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:51:46.206201 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.206112    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "a69bad32-fe1a-4556-8718-f8f3937d7dc1" (UID: "a69bad32-fe1a-4556-8718-f8f3937d7dc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:51:46.206201 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.206132    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-home" (OuterVolumeSpecName: "home") pod "a69bad32-fe1a-4556-8718-f8f3937d7dc1" (UID: "a69bad32-fe1a-4556-8718-f8f3937d7dc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:51:46.206361 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.206283    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:51:46.206361 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.206306    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:51:46.206361 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.206321    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:51:46.207974 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.207945    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kube-api-access-8jdp7" (OuterVolumeSpecName: "kube-api-access-8jdp7") pod "a69bad32-fe1a-4556-8718-f8f3937d7dc1" (UID: "a69bad32-fe1a-4556-8718-f8f3937d7dc1"). InnerVolumeSpecName "kube-api-access-8jdp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:51:46.208320 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.207975    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a69bad32-fe1a-4556-8718-f8f3937d7dc1" (UID: "a69bad32-fe1a-4556-8718-f8f3937d7dc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:51:46.208320 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.208090    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-dshm" (OuterVolumeSpecName: "dshm") pod "a69bad32-fe1a-4556-8718-f8f3937d7dc1" (UID: "a69bad32-fe1a-4556-8718-f8f3937d7dc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:51:46.242480 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.242450    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a69bad32-fe1a-4556-8718-f8f3937d7dc1" (UID: "a69bad32-fe1a-4556-8718-f8f3937d7dc1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:51:46.307316 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.307290    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jdp7\" (UniqueName: \"kubernetes.io/projected/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kube-api-access-8jdp7\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:51:46.307316 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.307316    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:51:46.307495 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.307327    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a69bad32-fe1a-4556-8718-f8f3937d7dc1-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:51:46.307495 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.307337    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69bad32-fe1a-4556-8718-f8f3937d7dc1-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:51:46.720447 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.720417    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4_a69bad32-fe1a-4556-8718-f8f3937d7dc1/storage-initializer/0.log"
Jun 02 16:51:46.720635 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.720461    2561 generic.go:358] "Generic (PLEG): container finished" podID="a69bad32-fe1a-4556-8718-f8f3937d7dc1" containerID="acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655" exitCode=137
Jun 02 16:51:46.720635 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.720530    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"
Jun 02 16:51:46.720635 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.720557    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4" event={"ID":"a69bad32-fe1a-4556-8718-f8f3937d7dc1","Type":"ContainerDied","Data":"acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655"}
Jun 02 16:51:46.720635 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.720599    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4" event={"ID":"a69bad32-fe1a-4556-8718-f8f3937d7dc1","Type":"ContainerDied","Data":"84e351838cdd77ea3f5a6635351612952247eb26d541b800cb4b3634fbdbb544"}
Jun 02 16:51:46.720635 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.720619    2561 scope.go:117] "RemoveContainer" containerID="acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655"
Jun 02 16:51:46.750386 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.750363    2561 scope.go:117] "RemoveContainer" containerID="acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655"
Jun 02 16:51:46.750756 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:51:46.750730    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655\": container with ID starting with acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655 not found: ID does not exist" containerID="acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655"
Jun 02 16:51:46.750868 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.750769    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655"} err="failed to get container status \"acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655\": rpc error: code = NotFound desc = could not find container \"acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655\": container with ID starting with acbf4a241864aed7d20cd1317f7847240ccfdb6c1865a31dd5d049c6f8ab6655 not found: ID does not exist"
Jun 02 16:51:46.759571 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.759546    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"]
Jun 02 16:51:46.765773 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:46.765748    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-02ac88e7-kserve-7c5cdd89cflrcd4"]
Jun 02 16:51:47.433518 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:47.433479    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a69bad32-fe1a-4556-8718-f8f3937d7dc1" path="/var/lib/kubelet/pods/a69bad32-fe1a-4556-8718-f8f3937d7dc1/volumes"
Jun 02 16:51:53.627879 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:53.627757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:51:53.634078 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:53.634060    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:51:53.636020 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:53.636003    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:51:53.641240 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:51:53.641221    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:52:21.854539 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:21.854462    2561 generic.go:358] "Generic (PLEG): container finished" podID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerID="e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c" exitCode=0
Jun 02 16:52:21.854539 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:21.854522    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1dc8002-9308-4bac-a2a5-477df9e2cce7","Type":"ContainerDied","Data":"e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c"}
Jun 02 16:52:22.859440 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:22.859403    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1dc8002-9308-4bac-a2a5-477df9e2cce7","Type":"ContainerStarted","Data":"af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de"}
Jun 02 16:52:22.880086 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:22.880023    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podStartSLOduration=58.880005743 podStartE2EDuration="58.880005743s" podCreationTimestamp="2026-06-02 16:51:24 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:52:22.878281919 +0000 UTC m=+3930.073061415" watchObservedRunningTime="2026-06-02 16:52:22.880005743 +0000 UTC m=+3930.074785248"
Jun 02 16:52:24.414451 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:24.414416    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:52:24.414451 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:24.414461    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:52:24.416060 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:24.416031    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:52:34.414770 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:34.414720    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:52:44.414729 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:44.414628    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:52:54.414403 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:52:54.414352    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:53:04.414069 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:53:04.414022    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:53:14.414678 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:53:14.414634    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:53:24.414788 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:53:24.414744    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:53:34.414484 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:53:34.414437    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:53:44.414463 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:53:44.414413    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.42:8000/health\": dial tcp 10.132.0.42:8000: connect: connection refused"
Jun 02 16:53:54.423319 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:53:54.423290    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:53:54.431239 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:53:54.431219    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:54:02.735359 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:02.735323    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jun 02 16:54:02.735836 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:02.735625    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main" containerID="cri-o://af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de" gracePeriod=30
Jun 02 16:54:03.484450 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.484428    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:54:03.617283 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.617202    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kserve-provision-location\") pod \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") "
Jun 02 16:54:03.617283 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.617240    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tmp-dir\") pod \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") "
Jun 02 16:54:03.617470 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.617284    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-dshm\") pod \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") "
Jun 02 16:54:03.617470 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.617307    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tls-certs\") pod \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") "
Jun 02 16:54:03.617470 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.617359    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvvqc\" (UniqueName: \"kubernetes.io/projected/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kube-api-access-fvvqc\") pod \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") "
Jun 02 16:54:03.617610 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.617526    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-model-cache\") pod \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") "
Jun 02 16:54:03.617610 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.617600    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-home\") pod \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\" (UID: \"b1dc8002-9308-4bac-a2a5-477df9e2cce7\") "
Jun 02 16:54:03.617789 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.617760    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1dc8002-9308-4bac-a2a5-477df9e2cce7" (UID: "b1dc8002-9308-4bac-a2a5-477df9e2cce7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:03.618341 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.618304    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.618467 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.618338    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-home" (OuterVolumeSpecName: "home") pod "b1dc8002-9308-4bac-a2a5-477df9e2cce7" (UID: "b1dc8002-9308-4bac-a2a5-477df9e2cce7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:03.619804 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.619780    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-dshm" (OuterVolumeSpecName: "dshm") pod "b1dc8002-9308-4bac-a2a5-477df9e2cce7" (UID: "b1dc8002-9308-4bac-a2a5-477df9e2cce7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:03.619994 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.619965    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1dc8002-9308-4bac-a2a5-477df9e2cce7" (UID: "b1dc8002-9308-4bac-a2a5-477df9e2cce7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:54:03.619994 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.619978    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kube-api-access-fvvqc" (OuterVolumeSpecName: "kube-api-access-fvvqc") pod "b1dc8002-9308-4bac-a2a5-477df9e2cce7" (UID: "b1dc8002-9308-4bac-a2a5-477df9e2cce7"). InnerVolumeSpecName "kube-api-access-fvvqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:54:03.631565 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.631537    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1dc8002-9308-4bac-a2a5-477df9e2cce7" (UID: "b1dc8002-9308-4bac-a2a5-477df9e2cce7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:03.679769 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.679735    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1dc8002-9308-4bac-a2a5-477df9e2cce7" (UID: "b1dc8002-9308-4bac-a2a5-477df9e2cce7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:54:03.719320 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.719291    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.719320 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.719314    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.719320 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.719324    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvvqc\" (UniqueName: \"kubernetes.io/projected/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kube-api-access-fvvqc\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.719522 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.719333    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.719522 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.719342    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:54:03.719522 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:03.719351    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dc8002-9308-4bac-a2a5-477df9e2cce7-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 16:54:04.239343 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.239311    2561 generic.go:358] "Generic (PLEG): container finished" podID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerID="af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de" exitCode=0
Jun 02 16:54:04.239767 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.239377    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"
Jun 02 16:54:04.239767 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.239391    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1dc8002-9308-4bac-a2a5-477df9e2cce7","Type":"ContainerDied","Data":"af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de"}
Jun 02 16:54:04.239767 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.239439    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0" event={"ID":"b1dc8002-9308-4bac-a2a5-477df9e2cce7","Type":"ContainerDied","Data":"ee3be610bdd2483eee7dcc37745038cae3227fdf556e4baf82a1734c159dd404"}
Jun 02 16:54:04.239767 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.239464    2561 scope.go:117] "RemoveContainer" containerID="af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de"
Jun 02 16:54:04.248548 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.248531    2561 scope.go:117] "RemoveContainer" containerID="e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c"
Jun 02 16:54:04.262429 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.262408    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jun 02 16:54:04.264082 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.264055    2561 scope.go:117] "RemoveContainer" containerID="af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de"
Jun 02 16:54:04.264415 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:54:04.264387    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de\": container with ID starting with af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de not found: ID does not exist" containerID="af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de"
Jun 02 16:54:04.264503 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.264426    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de"} err="failed to get container status \"af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de\": rpc error: code = NotFound desc = could not find container \"af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de\": container with ID starting with af9666010844598eca6132843d4423034a63a7ff0336186d3f91f97d33f7d3de not found: ID does not exist"
Jun 02 16:54:04.264503 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.264452    2561 scope.go:117] "RemoveContainer" containerID="e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c"
Jun 02 16:54:04.264732 ip-10-0-135-231 kubenswrapper[2561]: E0602 16:54:04.264714    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c\": container with ID starting with e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c not found: ID does not exist" containerID="e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c"
Jun 02 16:54:04.264816 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.264738    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c"} err="failed to get container status \"e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c\": rpc error: code = NotFound desc = could not find container \"e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c\": container with ID starting with e9a19055c8bf759efe17192817d0133971bcca6761aa66b4592f5c32a0f5185c not found: ID does not exist"
Jun 02 16:54:04.266018 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:04.265996    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0"]
Jun 02 16:54:05.434487 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:54:05.434449    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" path="/var/lib/kubelet/pods/b1dc8002-9308-4bac-a2a5-477df9e2cce7/volumes"
Jun 02 16:56:53.653366 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:56:53.653338    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:56:53.658916 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:56:53.658893    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 16:56:53.662034 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:56:53.662016    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 16:56:53.674378 ip-10-0-135-231 kubenswrapper[2561]: I0602 16:56:53.674351    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 17:01:53.679813 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:01:53.679720    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 17:01:53.685733 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:01:53.685711    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 17:01:53.694999 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:01:53.694980    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 17:01:53.700559 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:01:53.700542    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 17:04:59.107691 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:04:59.107606    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"]
Jun 02 17:04:59.108289 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:04:59.107993    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main" containerID="cri-o://f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda" gracePeriod=30
Jun 02 17:04:59.119312 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:04:59.119283    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"]
Jun 02 17:04:59.119665 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:04:59.119619    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar" containerID="cri-o://7745e9a205e76848e8bca75d92a5dc3c1c52bfaaf0b5b39b47c88afbaf4230c5" gracePeriod=30
Jun 02 17:04:59.119749 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:04:59.119646    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="storage-initializer" containerID="cri-o://5679a92aab0602f1ef8accb08d3d39d4a9c914b80d671bcfc9455609572eaabb" gracePeriod=30
Jun 02 17:04:59.514535 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:04:59.514503    2561 generic.go:358] "Generic (PLEG): container finished" podID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerID="7745e9a205e76848e8bca75d92a5dc3c1c52bfaaf0b5b39b47c88afbaf4230c5" exitCode=0
Jun 02 17:04:59.514699 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:04:59.514559    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" event={"ID":"e2d28bcf-1993-4bd0-87be-2215aa3970b0","Type":"ContainerDied","Data":"7745e9a205e76848e8bca75d92a5dc3c1c52bfaaf0b5b39b47c88afbaf4230c5"}
Jun 02 17:05:02.655375 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:02.655338    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jun 02 17:05:05.338287 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:05.338247    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jun 02 17:05:09.526849 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.526816    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"]
Jun 02 17:05:09.527373 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.527354    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a69bad32-fe1a-4556-8718-f8f3937d7dc1" containerName="storage-initializer"
Jun 02 17:05:09.527458 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.527376    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a69bad32-fe1a-4556-8718-f8f3937d7dc1" containerName="storage-initializer"
Jun 02 17:05:09.527458 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.527417    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main"
Jun 02 17:05:09.527458 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.527427    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main"
Jun 02 17:05:09.527458 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.527440    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="storage-initializer"
Jun 02 17:05:09.527458 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.527448    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="storage-initializer"
Jun 02 17:05:09.527715 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.527536    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="a69bad32-fe1a-4556-8718-f8f3937d7dc1" containerName="storage-initializer"
Jun 02 17:05:09.527715 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.527553    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="b1dc8002-9308-4bac-a2a5-477df9e2cce7" containerName="main"
Jun 02 17:05:09.530662 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.530641    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.533603 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.533573    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-v66wl\""
Jun 02 17:05:09.533603 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.533594    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jun 02 17:05:09.540840 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.540818    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"]
Jun 02 17:05:09.556574 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.556546    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"]
Jun 02 17:05:09.560575 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.560551    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.571643 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.571598    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"]
Jun 02 17:05:09.702291 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.702291 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702293    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-dshm\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.702546 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702313    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.702546 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702337    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.702546 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-home\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.702546 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702467    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tls-certs\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.702822 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.702822 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702641    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-home\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.702822 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f75bj\" (UniqueName: \"kubernetes.io/projected/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kube-api-access-f75bj\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.702822 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702739    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.702822 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702767    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.702822 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702818    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.703086 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702850    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94r2n\" (UniqueName: \"kubernetes.io/projected/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kube-api-access-94r2n\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.703086 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.702905    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-model-cache\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.803796 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.803720    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.803796 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.803772    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-home\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804003 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.803804    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f75bj\" (UniqueName: \"kubernetes.io/projected/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kube-api-access-f75bj\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804003 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.803837    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804003 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.803860    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804189 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804097    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804189 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804147    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-94r2n\" (UniqueName: \"kubernetes.io/projected/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kube-api-access-94r2n\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804316 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804200    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-home\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804316 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804144    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804316 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804224    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-model-cache\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804316 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804298    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804327    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-dshm\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804359    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804393    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804440    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-home\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804452    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tmp-dir\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804469    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tls-certs\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804490    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-model-cache\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.804851 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804513    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804851 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804733    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.804851 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804745    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-home\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.805011 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.804935    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.806609 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.806588    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.806712 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.806677    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.807010 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.806990    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-dshm\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.807281 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.807265    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tls-certs\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.811213 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.811194    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f75bj\" (UniqueName: \"kubernetes.io/projected/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kube-api-access-f75bj\") pod \"router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.811562 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.811521    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94r2n\" (UniqueName: \"kubernetes.io/projected/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kube-api-access-94r2n\") pod \"router-with-refs-pd-test-kserve-99479bc89-z7k5k\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.841011 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.840980    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:09.875841 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.875806    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:09.993524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.993323    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"]
Jun 02 17:05:09.997001 ip-10-0-135-231 kubenswrapper[2561]: W0602 17:05:09.996968    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a09746_0987_4e47_bd21_a8885a3c6d0e.slice/crio-372e0798c92b7f8784584c77c0231f44de9862d06359eaab7638fc3b4e813082 WatchSource:0}: Error finding container 372e0798c92b7f8784584c77c0231f44de9862d06359eaab7638fc3b4e813082: Status 404 returned error can't find the container with id 372e0798c92b7f8784584c77c0231f44de9862d06359eaab7638fc3b4e813082
Jun 02 17:05:09.999082 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:09.999067    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 17:05:10.018077 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:10.018055    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"]
Jun 02 17:05:10.019997 ip-10-0-135-231 kubenswrapper[2561]: W0602 17:05:10.019975    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8beedf1e_4e55_4d1a_b3e8_b1bd31d23c51.slice/crio-1dc14afebeb1772be17e0337a64f49ff6f754680dd5ce7a99686696de8d2191e WatchSource:0}: Error finding container 1dc14afebeb1772be17e0337a64f49ff6f754680dd5ce7a99686696de8d2191e: Status 404 returned error can't find the container with id 1dc14afebeb1772be17e0337a64f49ff6f754680dd5ce7a99686696de8d2191e
Jun 02 17:05:10.555210 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:10.555170    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" event={"ID":"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51","Type":"ContainerStarted","Data":"a3de489215bf21e12b5296ab3a05293d8bc38d9cc42991ff8064e9b079704001"}
Jun 02 17:05:10.555673 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:10.555224    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" event={"ID":"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51","Type":"ContainerStarted","Data":"1dc14afebeb1772be17e0337a64f49ff6f754680dd5ce7a99686696de8d2191e"}
Jun 02 17:05:10.556709 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:10.556684    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" event={"ID":"a2a09746-0987-4e47-bd21-a8885a3c6d0e","Type":"ContainerStarted","Data":"2a6249e9d2049992dc490a80cb3200d3d141cc8a4be9e4970f2dc9b2ddaf1f5b"}
Jun 02 17:05:10.556822 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:10.556715    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" event={"ID":"a2a09746-0987-4e47-bd21-a8885a3c6d0e","Type":"ContainerStarted","Data":"372e0798c92b7f8784584c77c0231f44de9862d06359eaab7638fc3b4e813082"}
Jun 02 17:05:10.556872 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:10.556815    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:11.570156 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:11.570029    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" event={"ID":"a2a09746-0987-4e47-bd21-a8885a3c6d0e","Type":"ContainerStarted","Data":"ecebd2a1607425da3d220e9298fc5bf9cd62c1b7f05471ec9942b82fd1d21424"}
Jun 02 17:05:12.655412 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:12.655366    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jun 02 17:05:15.338298 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:15.338252    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jun 02 17:05:15.584154 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:15.584114    2561 generic.go:358] "Generic (PLEG): container finished" podID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerID="ecebd2a1607425da3d220e9298fc5bf9cd62c1b7f05471ec9942b82fd1d21424" exitCode=0
Jun 02 17:05:15.584354 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:15.584192    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" event={"ID":"a2a09746-0987-4e47-bd21-a8885a3c6d0e","Type":"ContainerDied","Data":"ecebd2a1607425da3d220e9298fc5bf9cd62c1b7f05471ec9942b82fd1d21424"}
Jun 02 17:05:16.590727 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:16.590693    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" event={"ID":"a2a09746-0987-4e47-bd21-a8885a3c6d0e","Type":"ContainerStarted","Data":"1b0216be57ffe4b93898ce3a5bd537fca8864310e5a1bddf098bd1b063ad72c1"}
Jun 02 17:05:16.617280 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:16.617216    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podStartSLOduration=7.617197525 podStartE2EDuration="7.617197525s" podCreationTimestamp="2026-06-02 17:05:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 17:05:16.614717707 +0000 UTC m=+4703.809497207" watchObservedRunningTime="2026-06-02 17:05:16.617197525 +0000 UTC m=+4703.811977026"
Jun 02 17:05:18.599930 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:18.599896    2561 generic.go:358] "Generic (PLEG): container finished" podID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerID="a3de489215bf21e12b5296ab3a05293d8bc38d9cc42991ff8064e9b079704001" exitCode=0
Jun 02 17:05:18.600378 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:18.599982    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" event={"ID":"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51","Type":"ContainerDied","Data":"a3de489215bf21e12b5296ab3a05293d8bc38d9cc42991ff8064e9b079704001"}
Jun 02 17:05:19.605491 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:19.605462    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" event={"ID":"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51","Type":"ContainerStarted","Data":"cb92a3aa4f5d5c0721c6d6b9a7211d738d31d92bae1a644fcff3c77f71ad33da"}
Jun 02 17:05:19.630085 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:19.630026    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podStartSLOduration=10.63000765 podStartE2EDuration="10.63000765s" podCreationTimestamp="2026-06-02 17:05:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 17:05:19.627417934 +0000 UTC m=+4706.822197449" watchObservedRunningTime="2026-06-02 17:05:19.63000765 +0000 UTC m=+4706.824787150"
Jun 02 17:05:19.841935 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:19.841894    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:19.841935 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:19.841944    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:19.843289 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:19.843255    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:05:19.876323 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:19.876241    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:19.876323 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:19.876295    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:05:19.877768 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:19.877734    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:05:22.655690 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:22.655643    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jun 02 17:05:22.656060 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:22.655715    2561 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 17:05:25.338133 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:25.338079    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jun 02 17:05:29.423538 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.423506    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 17:05:29.518510 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.518477    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-model-cache\") pod \"84105eba-2b85-400c-8627-7f6d6ce0157a\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") "
Jun 02 17:05:29.518721 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.518520    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-kserve-provision-location\") pod \"84105eba-2b85-400c-8627-7f6d6ce0157a\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") "
Jun 02 17:05:29.518721 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.518544    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-tmp-dir\") pod \"84105eba-2b85-400c-8627-7f6d6ce0157a\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") "
Jun 02 17:05:29.518721 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.518601    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn87s\" (UniqueName: \"kubernetes.io/projected/84105eba-2b85-400c-8627-7f6d6ce0157a-kube-api-access-tn87s\") pod \"84105eba-2b85-400c-8627-7f6d6ce0157a\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") "
Jun 02 17:05:29.518721 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.518668    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84105eba-2b85-400c-8627-7f6d6ce0157a-tls-certs\") pod \"84105eba-2b85-400c-8627-7f6d6ce0157a\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") "
Jun 02 17:05:29.518721 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.518709    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-dshm\") pod \"84105eba-2b85-400c-8627-7f6d6ce0157a\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") "
Jun 02 17:05:29.519013 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.518733    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-home\") pod \"84105eba-2b85-400c-8627-7f6d6ce0157a\" (UID: \"84105eba-2b85-400c-8627-7f6d6ce0157a\") "
Jun 02 17:05:29.519013 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.518807    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-model-cache" (OuterVolumeSpecName: "model-cache") pod "84105eba-2b85-400c-8627-7f6d6ce0157a" (UID: "84105eba-2b85-400c-8627-7f6d6ce0157a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.519181 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.519144    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.519668 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.519639    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-home" (OuterVolumeSpecName: "home") pod "84105eba-2b85-400c-8627-7f6d6ce0157a" (UID: "84105eba-2b85-400c-8627-7f6d6ce0157a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.521837 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.521804    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84105eba-2b85-400c-8627-7f6d6ce0157a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84105eba-2b85-400c-8627-7f6d6ce0157a" (UID: "84105eba-2b85-400c-8627-7f6d6ce0157a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 17:05:29.521950 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.521932    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84105eba-2b85-400c-8627-7f6d6ce0157a-kube-api-access-tn87s" (OuterVolumeSpecName: "kube-api-access-tn87s") pod "84105eba-2b85-400c-8627-7f6d6ce0157a" (UID: "84105eba-2b85-400c-8627-7f6d6ce0157a"). InnerVolumeSpecName "kube-api-access-tn87s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 17:05:29.522036 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.522017    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-dshm" (OuterVolumeSpecName: "dshm") pod "84105eba-2b85-400c-8627-7f6d6ce0157a" (UID: "84105eba-2b85-400c-8627-7f6d6ce0157a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.529695 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.529670    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84105eba-2b85-400c-8627-7f6d6ce0157a" (UID: "84105eba-2b85-400c-8627-7f6d6ce0157a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.586402 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.586361    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "84105eba-2b85-400c-8627-7f6d6ce0157a" (UID: "84105eba-2b85-400c-8627-7f6d6ce0157a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.620644 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.620609    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84105eba-2b85-400c-8627-7f6d6ce0157a-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.620644 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.620639    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.620644 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.620648    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.620877 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.620658    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.620877 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.620667    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84105eba-2b85-400c-8627-7f6d6ce0157a-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.620877 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.620677    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn87s\" (UniqueName: \"kubernetes.io/projected/84105eba-2b85-400c-8627-7f6d6ce0157a-kube-api-access-tn87s\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.649202 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.649100    2561 generic.go:358] "Generic (PLEG): container finished" podID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerID="f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda" exitCode=137
Jun 02 17:05:29.649202 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.649183    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" event={"ID":"84105eba-2b85-400c-8627-7f6d6ce0157a","Type":"ContainerDied","Data":"f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda"}
Jun 02 17:05:29.649413 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.649216    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"
Jun 02 17:05:29.649413 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.649234    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm" event={"ID":"84105eba-2b85-400c-8627-7f6d6ce0157a","Type":"ContainerDied","Data":"ac94c97252209271084b865041ab1efdbc09075bcd4969cc04fd0cd71d0ca5d2"}
Jun 02 17:05:29.649413 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.649257    2561 scope.go:117] "RemoveContainer" containerID="f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda"
Jun 02 17:05:29.650799 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.650779    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9_e2d28bcf-1993-4bd0-87be-2215aa3970b0/storage-initializer/0.log"
Jun 02 17:05:29.651182 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.651138    2561 generic.go:358] "Generic (PLEG): container finished" podID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerID="5679a92aab0602f1ef8accb08d3d39d4a9c914b80d671bcfc9455609572eaabb" exitCode=137
Jun 02 17:05:29.651281 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.651187    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" event={"ID":"e2d28bcf-1993-4bd0-87be-2215aa3970b0","Type":"ContainerDied","Data":"5679a92aab0602f1ef8accb08d3d39d4a9c914b80d671bcfc9455609572eaabb"}
Jun 02 17:05:29.659480 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.659460    2561 scope.go:117] "RemoveContainer" containerID="187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b"
Jun 02 17:05:29.678048 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.677904    2561 scope.go:117] "RemoveContainer" containerID="f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda"
Jun 02 17:05:29.678136 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.678040    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"]
Jun 02 17:05:29.679840 ip-10-0-135-231 kubenswrapper[2561]: E0602 17:05:29.678414    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda\": container with ID starting with f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda not found: ID does not exist" containerID="f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda"
Jun 02 17:05:29.679840 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.678452    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda"} err="failed to get container status \"f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda\": rpc error: code = NotFound desc = could not find container \"f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda\": container with ID starting with f8bfe449864f4f88ed4e4c1b5f2b4c9e8d60ebdd5e52142c9a7a096a0cbbdfda not found: ID does not exist"
Jun 02 17:05:29.679840 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.678478    2561 scope.go:117] "RemoveContainer" containerID="187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b"
Jun 02 17:05:29.680113 ip-10-0-135-231 kubenswrapper[2561]: E0602 17:05:29.679918    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b\": container with ID starting with 187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b not found: ID does not exist" containerID="187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b"
Jun 02 17:05:29.680113 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.679949    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b"} err="failed to get container status \"187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b\": rpc error: code = NotFound desc = could not find container \"187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b\": container with ID starting with 187a19e563838f699c754898a93143b5d0cf5d5684826ce3443998ec96b0646b not found: ID does not exist"
Jun 02 17:05:29.681074 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.681049    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-5468649b76-9jhnm"]
Jun 02 17:05:29.807620 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.805484    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9_e2d28bcf-1993-4bd0-87be-2215aa3970b0/storage-initializer/0.log"
Jun 02 17:05:29.807620 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.806093    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 17:05:29.842173 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.842131    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:05:29.854596 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.854574    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:05:29.876553 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.876527    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:05:29.924096 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924023    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-home\") pod \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") "
Jun 02 17:05:29.924096 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924072    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhrwz\" (UniqueName: \"kubernetes.io/projected/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kube-api-access-hhrwz\") pod \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") "
Jun 02 17:05:29.924328 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924128    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-model-cache\") pod \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") "
Jun 02 17:05:29.924328 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924145    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-dshm\") pod \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") "
Jun 02 17:05:29.924328 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924192    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tls-certs\") pod \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") "
Jun 02 17:05:29.924328 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924236    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kserve-provision-location\") pod \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") "
Jun 02 17:05:29.924328 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924272    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tmp-dir\") pod \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\" (UID: \"e2d28bcf-1993-4bd0-87be-2215aa3970b0\") "
Jun 02 17:05:29.924590 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924350    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-home" (OuterVolumeSpecName: "home") pod "e2d28bcf-1993-4bd0-87be-2215aa3970b0" (UID: "e2d28bcf-1993-4bd0-87be-2215aa3970b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.924650 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924589    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2d28bcf-1993-4bd0-87be-2215aa3970b0" (UID: "e2d28bcf-1993-4bd0-87be-2215aa3970b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.924701 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924672    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2d28bcf-1993-4bd0-87be-2215aa3970b0" (UID: "e2d28bcf-1993-4bd0-87be-2215aa3970b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.924701 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924679    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.924802 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.924718    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:29.927011 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.926989    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2d28bcf-1993-4bd0-87be-2215aa3970b0" (UID: "e2d28bcf-1993-4bd0-87be-2215aa3970b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 17:05:29.927205 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.927182    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-dshm" (OuterVolumeSpecName: "dshm") pod "e2d28bcf-1993-4bd0-87be-2215aa3970b0" (UID: "e2d28bcf-1993-4bd0-87be-2215aa3970b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:29.927377 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.927354    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kube-api-access-hhrwz" (OuterVolumeSpecName: "kube-api-access-hhrwz") pod "e2d28bcf-1993-4bd0-87be-2215aa3970b0" (UID: "e2d28bcf-1993-4bd0-87be-2215aa3970b0"). InnerVolumeSpecName "kube-api-access-hhrwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 17:05:29.943144 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:29.943116    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2d28bcf-1993-4bd0-87be-2215aa3970b0" (UID: "e2d28bcf-1993-4bd0-87be-2215aa3970b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:05:30.025235 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.025203    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:30.025235 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.025236    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhrwz\" (UniqueName: \"kubernetes.io/projected/e2d28bcf-1993-4bd0-87be-2215aa3970b0-kube-api-access-hhrwz\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:30.025235 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.025251    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:30.025458 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.025265    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2d28bcf-1993-4bd0-87be-2215aa3970b0-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:30.025458 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.025275    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2d28bcf-1993-4bd0-87be-2215aa3970b0-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:05:30.658534 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.658505    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9_e2d28bcf-1993-4bd0-87be-2215aa3970b0/storage-initializer/0.log"
Jun 02 17:05:30.659020 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.658879    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9" event={"ID":"e2d28bcf-1993-4bd0-87be-2215aa3970b0","Type":"ContainerDied","Data":"bf4fcc5cb5ab9bdf6e881364b62a4fcbc92dc6d917e93b8093449bf8e0db4874"}
Jun 02 17:05:30.659020 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.658927    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"
Jun 02 17:05:30.659150 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.658928    2561 scope.go:117] "RemoveContainer" containerID="5679a92aab0602f1ef8accb08d3d39d4a9c914b80d671bcfc9455609572eaabb"
Jun 02 17:05:30.682325 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.682306    2561 scope.go:117] "RemoveContainer" containerID="7745e9a205e76848e8bca75d92a5dc3c1c52bfaaf0b5b39b47c88afbaf4230c5"
Jun 02 17:05:30.708908 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.702380    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"]
Jun 02 17:05:30.708908 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:30.706716    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-69f55f6745-nnxn9"]
Jun 02 17:05:31.433328 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:31.433296    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" path="/var/lib/kubelet/pods/84105eba-2b85-400c-8627-7f6d6ce0157a/volumes"
Jun 02 17:05:31.433748 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:31.433735    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" path="/var/lib/kubelet/pods/e2d28bcf-1993-4bd0-87be-2215aa3970b0/volumes"
Jun 02 17:05:39.841692 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:39.841646    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:05:39.876538 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:39.876485    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:05:49.841861 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:49.841810    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:05:49.876791 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:49.876747    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:05:59.841616 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:59.841562    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:05:59.876711 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:05:59.876669    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:06:09.842281 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:09.842154    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:06:09.876477 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:09.876432    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:06:19.841402 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:19.841350    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:06:19.876876 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:19.876832    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:06:29.842318 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:29.842255    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:06:29.876881 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:29.876840    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:06:39.841643 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:39.841595    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:06:39.876396 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:39.876355    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:06:49.841784 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:49.841734    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:06:49.876875 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:49.876833    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:06:53.707528 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:53.707426    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 17:06:53.714294 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:53.714272    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 17:06:53.723674 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:53.723656    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 17:06:53.732630 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:53.732611    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 17:06:59.842027 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:59.841979    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:06:59.877042 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:06:59.876995    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:07:09.841840 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:09.841779    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:07:09.876502 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:09.876464    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:07:19.842007 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:19.841951    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8001/health\": dial tcp 10.132.0.43:8001: connect: connection refused"
Jun 02 17:07:19.877125 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:19.877069    2561 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" probeResult="failure" output="Get \"https://10.132.0.44:8000/health\": dial tcp 10.132.0.44:8000: connect: connection refused"
Jun 02 17:07:29.851014 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:29.850987    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:07:29.862848 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:29.862821    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:07:29.888607 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:29.888575    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:07:29.896237 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:29.896214    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:07:42.032394 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:42.032358    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"]
Jun 02 17:07:42.033350 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:42.032738    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main" containerID="cri-o://cb92a3aa4f5d5c0721c6d6b9a7211d738d31d92bae1a644fcff3c77f71ad33da" gracePeriod=30
Jun 02 17:07:42.035065 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:42.035042    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"]
Jun 02 17:07:42.035512 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:42.035452    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main" containerID="cri-o://1b0216be57ffe4b93898ce3a5bd537fca8864310e5a1bddf098bd1b063ad72c1" gracePeriod=30
Jun 02 17:07:59.435777 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:59.435742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:07:59.449953 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:59.449921    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:07:59.460097 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:59.460072    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:07:59.479095 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:59.479061    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:07:59.485751 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:59.485726    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:07:59.509805 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:07:59.509781    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:00.535837 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:00.535809    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:00.543061 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:00.543036    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:00.554010 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:00.553983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:00.573575 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:00.573553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:00.580509 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:00.580493    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:00.607215 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:00.607196    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:01.661477 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:01.661445    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:01.668086 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:01.668064    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:01.679152 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:01.679132    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:01.702140 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:01.702110    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:01.709680 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:01.709649    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:01.734138 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:01.734114    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:02.773748 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:02.773716    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:02.782599 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:02.782575    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:02.796935 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:02.796911    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:02.828571 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:02.828550    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:02.846588 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:02.846561    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:02.874757 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:02.874736    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:03.889106 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:03.889072    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:03.894994 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:03.894972    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:03.904465 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:03.904446    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:03.922998 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:03.922973    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:03.928937 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:03.928915    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:03.952563 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:03.952533    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:04.960277 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:04.960244    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:04.966585 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:04.966563    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:04.976698 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:04.976678    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:04.995149 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:04.995103    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:05.001417 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:05.001399    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:05.030137 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:05.030117    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:06.057637 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:06.057604    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:06.067782 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:06.067759    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:06.078314 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:06.078295    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:06.097360 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:06.097338    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:06.104302 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:06.104279    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:06.128746 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:06.128719    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:07.162565 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:07.162526    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:07.169077 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:07.169057    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:07.179986 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:07.179961    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:07.199878 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:07.199854    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:07.206759 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:07.206741    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:07.235640 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:07.235617    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:08.276859 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:08.276829    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:08.291339 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:08.291314    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:08.301808 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:08.301785    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:08.321657 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:08.321632    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:08.332872 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:08.332853    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:08.358122 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:08.358102    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:09.451692 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:09.451655    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:09.463384 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:09.463360    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:09.472988 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:09.472966    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:09.491626 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:09.491606    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:09.502806 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:09.502784    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:09.526611 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:09.526591    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:10.625312 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:10.625281    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:10.640555 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:10.640531    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:10.650784 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:10.650764    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:10.673189 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:10.673127    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:10.679749 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:10.679731    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:10.708077 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:10.708051    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:11.817372 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:11.817335    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:11.831727 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:11.831700    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:11.841531 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:11.841509    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:11.860659 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:11.860634    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:11.867316 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:11.867297    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:11.891952 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:11.891928    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:12.036240 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.036203    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="llm-d-routing-sidecar" containerID="cri-o://2a6249e9d2049992dc490a80cb3200d3d141cc8a4be9e4970f2dc9b2ddaf1f5b" gracePeriod=2
Jun 02 17:08:12.268602 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.268580    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:12.269573 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.269542    2561 generic.go:358] "Generic (PLEG): container finished" podID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerID="1b0216be57ffe4b93898ce3a5bd537fca8864310e5a1bddf098bd1b063ad72c1" exitCode=137
Jun 02 17:08:12.269573 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.269569    2561 generic.go:358] "Generic (PLEG): container finished" podID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerID="2a6249e9d2049992dc490a80cb3200d3d141cc8a4be9e4970f2dc9b2ddaf1f5b" exitCode=0
Jun 02 17:08:12.269748 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.269609    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" event={"ID":"a2a09746-0987-4e47-bd21-a8885a3c6d0e","Type":"ContainerDied","Data":"1b0216be57ffe4b93898ce3a5bd537fca8864310e5a1bddf098bd1b063ad72c1"}
Jun 02 17:08:12.269748 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.269649    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" event={"ID":"a2a09746-0987-4e47-bd21-a8885a3c6d0e","Type":"ContainerDied","Data":"2a6249e9d2049992dc490a80cb3200d3d141cc8a4be9e4970f2dc9b2ddaf1f5b"}
Jun 02 17:08:12.271664 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.271637    2561 generic.go:358] "Generic (PLEG): container finished" podID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerID="cb92a3aa4f5d5c0721c6d6b9a7211d738d31d92bae1a644fcff3c77f71ad33da" exitCode=137
Jun 02 17:08:12.271752 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.271681    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" event={"ID":"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51","Type":"ContainerDied","Data":"cb92a3aa4f5d5c0721c6d6b9a7211d738d31d92bae1a644fcff3c77f71ad33da"}
Jun 02 17:08:12.339518 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.339497    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:12.340272 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.340251    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:08:12.342936 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.342919    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:08:12.420926 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.420899    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tls-certs\") pod \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") "
Jun 02 17:08:12.421097 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.420937    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-home\") pod \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") "
Jun 02 17:08:12.421097 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.420959    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kserve-provision-location\") pod \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") "
Jun 02 17:08:12.421097 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.420982    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-dshm\") pod \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") "
Jun 02 17:08:12.421097 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421010    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94r2n\" (UniqueName: \"kubernetes.io/projected/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kube-api-access-94r2n\") pod \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") "
Jun 02 17:08:12.421097 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421033    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tmp-dir\") pod \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") "
Jun 02 17:08:12.421097 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421069    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tmp-dir\") pod \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") "
Jun 02 17:08:12.421368 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421113    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f75bj\" (UniqueName: \"kubernetes.io/projected/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kube-api-access-f75bj\") pod \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") "
Jun 02 17:08:12.421368 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421141    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tls-certs\") pod \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") "
Jun 02 17:08:12.421368 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421192    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-model-cache\") pod \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") "
Jun 02 17:08:12.421368 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421240    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kserve-provision-location\") pod \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") "
Jun 02 17:08:12.421368 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421271    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-model-cache\") pod \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") "
Jun 02 17:08:12.421368 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421318    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-dshm\") pod \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\" (UID: \"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51\") "
Jun 02 17:08:12.421606 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421371    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-home\") pod \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\" (UID: \"a2a09746-0987-4e47-bd21-a8885a3c6d0e\") "
Jun 02 17:08:12.421606 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421590    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-home" (OuterVolumeSpecName: "home") pod "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" (UID: "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.421711 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421690    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.421896 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.421875    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-model-cache" (OuterVolumeSpecName: "model-cache") pod "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" (UID: "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.423852 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.422539    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2a09746-0987-4e47-bd21-a8885a3c6d0e" (UID: "a2a09746-0987-4e47-bd21-a8885a3c6d0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.423968 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.423914    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-home" (OuterVolumeSpecName: "home") pod "a2a09746-0987-4e47-bd21-a8885a3c6d0e" (UID: "a2a09746-0987-4e47-bd21-a8885a3c6d0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.424480 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.424448    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" (UID: "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 17:08:12.424686 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.424652    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2a09746-0987-4e47-bd21-a8885a3c6d0e" (UID: "a2a09746-0987-4e47-bd21-a8885a3c6d0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 17:08:12.425351 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.425252    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-dshm" (OuterVolumeSpecName: "dshm") pod "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" (UID: "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.425351 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.425316    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-dshm" (OuterVolumeSpecName: "dshm") pod "a2a09746-0987-4e47-bd21-a8885a3c6d0e" (UID: "a2a09746-0987-4e47-bd21-a8885a3c6d0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.425504 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.425470    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kube-api-access-f75bj" (OuterVolumeSpecName: "kube-api-access-f75bj") pod "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" (UID: "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51"). InnerVolumeSpecName "kube-api-access-f75bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 17:08:12.425935 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.425911    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kube-api-access-94r2n" (OuterVolumeSpecName: "kube-api-access-94r2n") pod "a2a09746-0987-4e47-bd21-a8885a3c6d0e" (UID: "a2a09746-0987-4e47-bd21-a8885a3c6d0e"). InnerVolumeSpecName "kube-api-access-94r2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 17:08:12.444622 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.444594    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" (UID: "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.444744 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.444666    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2a09746-0987-4e47-bd21-a8885a3c6d0e" (UID: "a2a09746-0987-4e47-bd21-a8885a3c6d0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.492221 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.492186    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2a09746-0987-4e47-bd21-a8885a3c6d0e" (UID: "a2a09746-0987-4e47-bd21-a8885a3c6d0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.492874 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.492852    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" (UID: "8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 17:08:12.522461 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522437    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522461 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522461    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522461 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522470    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522480    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94r2n\" (UniqueName: \"kubernetes.io/projected/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kube-api-access-94r2n\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522490    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522499    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tmp-dir\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522508    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f75bj\" (UniqueName: \"kubernetes.io/projected/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-kube-api-access-f75bj\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522516    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-tls-certs\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522525    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522533    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-kserve-provision-location\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522543    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-model-cache\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522554    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51-dshm\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.522666 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.522566    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2a09746-0987-4e47-bd21-a8885a3c6d0e-home\") on node \"ip-10-0-135-231.ec2.internal\" DevicePath \"\""
Jun 02 17:08:12.962520 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.962489    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:12.971093 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.971066    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/llm-d-routing-sidecar/0.log"
Jun 02 17:08:12.982668 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:12.982648    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/storage-initializer/0.log"
Jun 02 17:08:13.001844 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.001818    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/main/0.log"
Jun 02 17:08:13.009033 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.008990    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97_8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/storage-initializer/0.log"
Jun 02 17:08:13.040457 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.040434    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:13.277267 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.277193    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-with-refs-pd-test-kserve-99479bc89-z7k5k_a2a09746-0987-4e47-bd21-a8885a3c6d0e/main/0.log"
Jun 02 17:08:13.277870 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.277852    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"
Jun 02 17:08:13.277947 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.277849    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k" event={"ID":"a2a09746-0987-4e47-bd21-a8885a3c6d0e","Type":"ContainerDied","Data":"372e0798c92b7f8784584c77c0231f44de9862d06359eaab7638fc3b4e813082"}
Jun 02 17:08:13.278003 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.277968    2561 scope.go:117] "RemoveContainer" containerID="1b0216be57ffe4b93898ce3a5bd537fca8864310e5a1bddf098bd1b063ad72c1"
Jun 02 17:08:13.279418 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.279385    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97" event={"ID":"8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51","Type":"ContainerDied","Data":"1dc14afebeb1772be17e0337a64f49ff6f754680dd5ce7a99686696de8d2191e"}
Jun 02 17:08:13.279524 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.279427    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"
Jun 02 17:08:13.291780 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.291757    2561 scope.go:117] "RemoveContainer" containerID="ecebd2a1607425da3d220e9298fc5bf9cd62c1b7f05471ec9942b82fd1d21424"
Jun 02 17:08:13.302817 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.302799    2561 scope.go:117] "RemoveContainer" containerID="2a6249e9d2049992dc490a80cb3200d3d141cc8a4be9e4970f2dc9b2ddaf1f5b"
Jun 02 17:08:13.308611 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.308555    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"]
Jun 02 17:08:13.311648 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.311615    2561 scope.go:117] "RemoveContainer" containerID="cb92a3aa4f5d5c0721c6d6b9a7211d738d31d92bae1a644fcff3c77f71ad33da"
Jun 02 17:08:13.312858 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.312832    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-99479bc89-z7k5k"]
Jun 02 17:08:13.319918 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.319889    2561 scope.go:117] "RemoveContainer" containerID="a3de489215bf21e12b5296ab3a05293d8bc38d9cc42991ff8064e9b079704001"
Jun 02 17:08:13.322720 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.322700    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"]
Jun 02 17:08:13.327182 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.327144    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-59c7cbc884-8jl97"]
Jun 02 17:08:13.433118 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.433088    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" path="/var/lib/kubelet/pods/8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51/volumes"
Jun 02 17:08:13.433544 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:13.433531    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" path="/var/lib/kubelet/pods/a2a09746-0987-4e47-bd21-a8885a3c6d0e/volumes"
Jun 02 17:08:14.068494 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:14.068432    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_stop-feature-test-kserve-6bcb45f7c5-2lt4v_ffa2fa3f-ee94-4169-88d7-75768c2faa3a/storage-initializer/0.log"
Jun 02 17:08:15.147651 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:15.147619    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674954b4d5-mrfmn_906f54a9-b574-4bd6-b40d-e4383642b3ab/router/0.log"
Jun 02 17:08:15.956116 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:15.956084    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674954b4d5-mrfmn_906f54a9-b574-4bd6-b40d-e4383642b3ab/router/0.log"
Jun 02 17:08:16.773473 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:16.773445    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674954b4d5-mrfmn_906f54a9-b574-4bd6-b40d-e4383642b3ab/router/0.log"
Jun 02 17:08:17.627929 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:17.627891    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-hvtgk_8a6ae02a-3982-4eaa-9a42-8eda66b28778/manager/0.log"
Jun 02 17:08:17.636467 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:17.636445    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-rmp8h_4e6e797a-89ac-4ab6-8631-60ee75166155/limitador/0.log"
Jun 02 17:08:18.533959 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:18.533924    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-hvtgk_8a6ae02a-3982-4eaa-9a42-8eda66b28778/manager/0.log"
Jun 02 17:08:18.543532 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:18.543506    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-rmp8h_4e6e797a-89ac-4ab6-8631-60ee75166155/limitador/0.log"
Jun 02 17:08:19.464395 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:19.464352    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-hvtgk_8a6ae02a-3982-4eaa-9a42-8eda66b28778/manager/0.log"
Jun 02 17:08:19.472890 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:19.472870    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-rmp8h_4e6e797a-89ac-4ab6-8631-60ee75166155/limitador/0.log"
Jun 02 17:08:20.367279 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:20.367245    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-hvtgk_8a6ae02a-3982-4eaa-9a42-8eda66b28778/manager/0.log"
Jun 02 17:08:20.383996 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:20.383963    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-rmp8h_4e6e797a-89ac-4ab6-8631-60ee75166155/limitador/0.log"
Jun 02 17:08:21.280292 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:21.280256    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-hvtgk_8a6ae02a-3982-4eaa-9a42-8eda66b28778/manager/0.log"
Jun 02 17:08:21.293447 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:21.293423    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-rmp8h_4e6e797a-89ac-4ab6-8631-60ee75166155/limitador/0.log"
Jun 02 17:08:26.561233 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:26.561207    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pn7nc_f77022b6-028e-4650-b07e-48525a6590f2/global-pull-secret-syncer/0.log"
Jun 02 17:08:26.705176 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:26.705144    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-drrq4_cfc3ce25-2f60-4d5d-8e87-ef9f61e17e8e/konnectivity-agent/0.log"
Jun 02 17:08:26.791624 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:26.791593    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-231.ec2.internal_7e787f9b1152ee0a0c6c9b3174979975/haproxy/0.log"
Jun 02 17:08:31.065609 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:31.065524    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-hvtgk_8a6ae02a-3982-4eaa-9a42-8eda66b28778/manager/0.log"
Jun 02 17:08:31.084425 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:31.084397    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-67566c68b4-rmp8h_4e6e797a-89ac-4ab6-8631-60ee75166155/limitador/0.log"
Jun 02 17:08:31.997322 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:31.997292    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51232482-b19e-4fe8-ac02-2acaf7375f79/alertmanager/0.log"
Jun 02 17:08:32.022407 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.022382    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51232482-b19e-4fe8-ac02-2acaf7375f79/config-reloader/0.log"
Jun 02 17:08:32.044672 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.044644    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51232482-b19e-4fe8-ac02-2acaf7375f79/kube-rbac-proxy-web/0.log"
Jun 02 17:08:32.068724 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.068698    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51232482-b19e-4fe8-ac02-2acaf7375f79/kube-rbac-proxy/0.log"
Jun 02 17:08:32.094865 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.094787    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51232482-b19e-4fe8-ac02-2acaf7375f79/kube-rbac-proxy-metric/0.log"
Jun 02 17:08:32.119800 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.119777    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51232482-b19e-4fe8-ac02-2acaf7375f79/prom-label-proxy/0.log"
Jun 02 17:08:32.144199 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.144175    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_51232482-b19e-4fe8-ac02-2acaf7375f79/init-config-reloader/0.log"
Jun 02 17:08:32.556685 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.556646    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zmpcb_a1460ceb-a365-4964-871a-46ed94304692/node-exporter/0.log"
Jun 02 17:08:32.574871 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.574841    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zmpcb_a1460ceb-a365-4964-871a-46ed94304692/kube-rbac-proxy/0.log"
Jun 02 17:08:32.595149 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.595129    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zmpcb_a1460ceb-a365-4964-871a-46ed94304692/init-textfile/0.log"
Jun 02 17:08:32.724801 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.724777    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39d84e3d-5e9d-498a-8f07-bad27fcc492a/prometheus/0.log"
Jun 02 17:08:32.740411 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.740375    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39d84e3d-5e9d-498a-8f07-bad27fcc492a/config-reloader/0.log"
Jun 02 17:08:32.761148 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.761124    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39d84e3d-5e9d-498a-8f07-bad27fcc492a/thanos-sidecar/0.log"
Jun 02 17:08:32.781188 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.781151    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39d84e3d-5e9d-498a-8f07-bad27fcc492a/kube-rbac-proxy-web/0.log"
Jun 02 17:08:32.802286 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.802266    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39d84e3d-5e9d-498a-8f07-bad27fcc492a/kube-rbac-proxy/0.log"
Jun 02 17:08:32.825897 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.825865    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39d84e3d-5e9d-498a-8f07-bad27fcc492a/kube-rbac-proxy-thanos/0.log"
Jun 02 17:08:32.858314 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.858296    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39d84e3d-5e9d-498a-8f07-bad27fcc492a/init-config-reloader/0.log"
Jun 02 17:08:32.897577 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.897549    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-658bcd8bdf-sqwbg_4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c/prometheus-operator/0.log"
Jun 02 17:08:32.921742 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.921713    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-658bcd8bdf-sqwbg_4d8e2eb9-c80f-487e-a35b-7d76dd4ab47c/kube-rbac-proxy/0.log"
Jun 02 17:08:32.996397 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:32.996367    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54b7b45c56-cgqrn_25210a6c-f667-42dd-8b97-1a4fda7ab373/telemeter-client/0.log"
Jun 02 17:08:33.021081 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:33.021058    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54b7b45c56-cgqrn_25210a6c-f667-42dd-8b97-1a4fda7ab373/reload/0.log"
Jun 02 17:08:33.044729 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:33.044703    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54b7b45c56-cgqrn_25210a6c-f667-42dd-8b97-1a4fda7ab373/kube-rbac-proxy/0.log"
Jun 02 17:08:34.869426 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:34.869394    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/1.log"
Jun 02 17:08:34.878288 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:34.878259    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-86d7c6cc77-c4r5l_7b8caa37-855d-4680-9fff-34dc85906216/console-operator/2.log"
Jun 02 17:08:35.446006 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.445975    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"]
Jun 02 17:08:35.446320 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446307    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="storage-initializer"
Jun 02 17:08:35.446376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446322    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="storage-initializer"
Jun 02 17:08:35.446376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446337    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main"
Jun 02 17:08:35.446376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446343    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main"
Jun 02 17:08:35.446376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446351    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="storage-initializer"
Jun 02 17:08:35.446376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446357    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="storage-initializer"
Jun 02 17:08:35.446376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446365    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main"
Jun 02 17:08:35.446376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446370    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main"
Jun 02 17:08:35.446376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446377    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="storage-initializer"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446382    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="storage-initializer"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446390    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446395    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446402    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="storage-initializer"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446407    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="storage-initializer"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446413    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446418    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446425    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="llm-d-routing-sidecar"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446429    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="llm-d-routing-sidecar"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446489    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="84105eba-2b85-400c-8627-7f6d6ce0157a" containerName="main"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446498    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="llm-d-routing-sidecar"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446505    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="llm-d-routing-sidecar"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446511    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="e2d28bcf-1993-4bd0-87be-2215aa3970b0" containerName="storage-initializer"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446517    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="a2a09746-0987-4e47-bd21-a8885a3c6d0e" containerName="main"
Jun 02 17:08:35.446612 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.446526    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8beedf1e-4e55-4d1a-b3e8-b1bd31d23c51" containerName="main"
Jun 02 17:08:35.451194 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.451156    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.454109 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.453925    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nxgdw\"/\"default-dockercfg-xs2rl\""
Jun 02 17:08:35.454109 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.453977    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nxgdw\"/\"openshift-service-ca.crt\""
Jun 02 17:08:35.455094 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.455062    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nxgdw\"/\"kube-root-ca.crt\""
Jun 02 17:08:35.457046 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.457025    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"]
Jun 02 17:08:35.626298 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.626258    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpst4\" (UniqueName: \"kubernetes.io/projected/23917495-4f54-42ab-bb32-0ded5006f3b3-kube-api-access-mpst4\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.626478 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.626320    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-sys\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.626478 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.626352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-lib-modules\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.626478 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.626380    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-proc\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.626478 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.626435    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-podres\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727470 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727392    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-sys\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727470 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727442    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-lib-modules\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727470 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727470    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-proc\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727650 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727486    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-podres\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727650 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727503    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-sys\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727650 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727531    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mpst4\" (UniqueName: \"kubernetes.io/projected/23917495-4f54-42ab-bb32-0ded5006f3b3-kube-api-access-mpst4\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727650 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727560    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-proc\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727650 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727623    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-lib-modules\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.727650 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.727636    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/23917495-4f54-42ab-bb32-0ded5006f3b3-podres\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.735376 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.735356    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpst4\" (UniqueName: \"kubernetes.io/projected/23917495-4f54-42ab-bb32-0ded5006f3b3-kube-api-access-mpst4\") pod \"perf-node-gather-daemonset-ddfjf\" (UID: \"23917495-4f54-42ab-bb32-0ded5006f3b3\") " pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.762485 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.762438    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:35.887956 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:35.887929    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"]
Jun 02 17:08:35.890171 ip-10-0-135-231 kubenswrapper[2561]: W0602 17:08:35.890122    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod23917495_4f54_42ab_bb32_0ded5006f3b3.slice/crio-221ff1ad276f305a63cdb1ce2eeed1412954b6adcf3ecda53f8775d103d3ad94 WatchSource:0}: Error finding container 221ff1ad276f305a63cdb1ce2eeed1412954b6adcf3ecda53f8775d103d3ad94: Status 404 returned error can't find the container with id 221ff1ad276f305a63cdb1ce2eeed1412954b6adcf3ecda53f8775d103d3ad94
Jun 02 17:08:36.362243 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:36.362113    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf" event={"ID":"23917495-4f54-42ab-bb32-0ded5006f3b3","Type":"ContainerStarted","Data":"01d5768f101207df3a53ae15e2231726aa660375f2f4cc9fa6d2dc2490c1d3c2"}
Jun 02 17:08:36.362243 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:36.362148    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf" event={"ID":"23917495-4f54-42ab-bb32-0ded5006f3b3","Type":"ContainerStarted","Data":"221ff1ad276f305a63cdb1ce2eeed1412954b6adcf3ecda53f8775d103d3ad94"}
Jun 02 17:08:36.362411 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:36.362262    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:36.378074 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:36.378030    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf" podStartSLOduration=1.378016725 podStartE2EDuration="1.378016725s" podCreationTimestamp="2026-06-02 17:08:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 17:08:36.376219993 +0000 UTC m=+4903.570999491" watchObservedRunningTime="2026-06-02 17:08:36.378016725 +0000 UTC m=+4903.572796224"
Jun 02 17:08:36.518948 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:36.518919    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7tj9z_5dbb0410-abce-4622-bd7c-dd7ed0faa3b5/dns/0.log"
Jun 02 17:08:36.536298 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:36.536272    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7tj9z_5dbb0410-abce-4622-bd7c-dd7ed0faa3b5/kube-rbac-proxy/0.log"
Jun 02 17:08:36.666145 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:36.666117    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-66lpd_1f7581f7-059f-4573-aec7-312b2de9f73c/dns-node-resolver/0.log"
Jun 02 17:08:37.214530 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:37.214503    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zqnmk_5948a580-09d3-4717-aa41-4ed1de678f0d/node-ca/0.log"
Jun 02 17:08:38.095085 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:38.095049    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674954b4d5-mrfmn_906f54a9-b574-4bd6-b40d-e4383642b3ab/router/0.log"
Jun 02 17:08:38.573678 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:38.573644    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w2q49_a1ba04ba-dd8d-4f15-9629-f84b2f6e81dc/serve-healthcheck-canary/0.log"
Jun 02 17:08:39.013416 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:39.013384    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-788bf58df5-lv6l8_8ca99b2d-46f2-454b-89eb-df55517239d0/insights-operator/0.log"
Jun 02 17:08:39.015122 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:39.015098    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-788bf58df5-lv6l8_8ca99b2d-46f2-454b-89eb-df55517239d0/insights-operator/1.log"
Jun 02 17:08:39.112231 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:39.112203    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drq7l_5c7930c7-89b3-4575-ba9a-f8dac8c9e87f/kube-rbac-proxy/0.log"
Jun 02 17:08:39.130921 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:39.130892    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drq7l_5c7930c7-89b3-4575-ba9a-f8dac8c9e87f/exporter/0.log"
Jun 02 17:08:39.150381 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:39.150358    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-drq7l_5c7930c7-89b3-4575-ba9a-f8dac8c9e87f/extractor/0.log"
Jun 02 17:08:41.716896 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:41.716865    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74c68556d6-bshb6_9fe3052e-1694-446e-9f09-607c2f07f746/manager/0.log"
Jun 02 17:08:42.375610 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:42.375586    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nxgdw/perf-node-gather-daemonset-ddfjf"
Jun 02 17:08:47.509741 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:47.509697    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-96ddc787c-pn6mk_64f59933-6166-4e8c-b115-ac029db80096/kube-storage-version-migrator-operator/1.log"
Jun 02 17:08:47.511729 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:47.511698    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-96ddc787c-pn6mk_64f59933-6166-4e8c-b115-ac029db80096/kube-storage-version-migrator-operator/0.log"
Jun 02 17:08:48.429040 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.429001    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8htr9_20a616f5-634d-4520-9fc3-5fe20beac101/kube-multus-additional-cni-plugins/0.log"
Jun 02 17:08:48.450185 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.450135    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8htr9_20a616f5-634d-4520-9fc3-5fe20beac101/egress-router-binary-copy/0.log"
Jun 02 17:08:48.468603 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.468582    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8htr9_20a616f5-634d-4520-9fc3-5fe20beac101/cni-plugins/0.log"
Jun 02 17:08:48.487345 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.487312    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8htr9_20a616f5-634d-4520-9fc3-5fe20beac101/bond-cni-plugin/0.log"
Jun 02 17:08:48.506602 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.506585    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8htr9_20a616f5-634d-4520-9fc3-5fe20beac101/routeoverride-cni/0.log"
Jun 02 17:08:48.525531 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.525512    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8htr9_20a616f5-634d-4520-9fc3-5fe20beac101/whereabouts-cni-bincopy/0.log"
Jun 02 17:08:48.547214 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.547193    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-8htr9_20a616f5-634d-4520-9fc3-5fe20beac101/whereabouts-cni/0.log"
Jun 02 17:08:48.902126 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.902097    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-qqwfk_67e734df-160b-4056-8cfa-811074a6df62/kube-multus/0.log"
Jun 02 17:08:48.978230 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.978199    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-48g4z_6d86be44-4451-4903-bf56-5179cec7698d/network-metrics-daemon/0.log"
Jun 02 17:08:48.995445 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:48.995413    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-48g4z_6d86be44-4451-4903-bf56-5179cec7698d/kube-rbac-proxy/0.log"
Jun 02 17:08:50.176976 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.176946    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-controller/0.log"
Jun 02 17:08:50.191880 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.191853    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/0.log"
Jun 02 17:08:50.234604 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.234578    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovn-acl-logging/1.log"
Jun 02 17:08:50.256550 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.256521    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/kube-rbac-proxy-node/0.log"
Jun 02 17:08:50.275985 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.275964    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/kube-rbac-proxy-ovn-metrics/0.log"
Jun 02 17:08:50.291199 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.291028    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/northd/0.log"
Jun 02 17:08:50.337203 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.337178    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/nbdb/0.log"
Jun 02 17:08:50.356763 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.356744    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/sbdb/0.log"
Jun 02 17:08:50.575708 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:50.575626    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hjmpt_1f5684ac-ab95-4d17-9b1f-73d3bb1c8560/ovnkube-controller/0.log"
Jun 02 17:08:51.934000 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:51.933969    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-56d84b9f44-b2zdc_c18af438-7ff0-4e15-83a0-320a446f2bd3/check-endpoints/0.log"
Jun 02 17:08:52.004446 ip-10-0-135-231 kubenswrapper[2561]: I0602 17:08:52.004418    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rmnx9_5467a93a-b318-4861-b91b-11becab47db8/network-check-target-container/0.log"