Jun 02 15:46:35.537683 ip-10-0-133-168 systemd[1]: Starting Kubernetes Kubelet...
Jun 02 15:46:35.974036 ip-10-0-133-168 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:35.974036 ip-10-0-133-168 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 02 15:46:35.974036 ip-10-0-133-168 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:35.974036 ip-10-0-133-168 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 02 15:46:35.974036 ip-10-0-133-168 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 02 15:46:35.975593 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.975506    2577 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 02 15:46:35.982021 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.981994    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:35.982021 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982014    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:35.982021 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982020    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:35.982021 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982023    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:35.982021 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982027    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982032    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982036    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982044    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982049    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982053    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982056    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982060    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982064    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982069    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982074    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982079    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982083    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982087    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982090    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982094    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982098    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982102    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:35.982311 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982105    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982109    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982113    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982118    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982122    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982125    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982129    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982133    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982137    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982141    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982145    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982149    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982152    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982156    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982161    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982165    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982171    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982175    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982179    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982184    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:35.983042 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982188    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982192    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982196    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982200    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982206    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982211    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982215    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982219    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982223    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982229    2577 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982233    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982237    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982241    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982245    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982250    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982254    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982258    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982262    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982266    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982270    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:35.983766 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982275    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982279    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982283    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982287    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982291    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982295    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982299    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982303    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982308    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982313    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982317    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982321    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982326    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982329    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982336    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982340    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982347    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982352    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982356    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982360    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:35.984328 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982364    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982369    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982373    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.982376    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983008    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983017    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983022    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983026    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983033    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983038    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983042    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983047    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983052    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983056    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983060    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983064    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983068    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983072    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983077    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983080    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:35.985153 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983085    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983090    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983094    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983099    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983103    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983115    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983119    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983124    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983130    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983134    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983138    2577 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983142    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983146    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983150    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983154    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983158    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983162    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983166    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983170    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983175    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:35.985847 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983179    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983183    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983188    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983192    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983197    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983201    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983205    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983209    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983213    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983217    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983221    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983226    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983230    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983234    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983238    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983243    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983246    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983250    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983255    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983259    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:35.986368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983263    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983268    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983272    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983276    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983281    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983287    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983292    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983297    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983301    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983305    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983309    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983313    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983317    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983321    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983327    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983331    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983336    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983341    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983347    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:35.986989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983353    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983357    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983361    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983365    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983371    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983376    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983380    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983384    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983404    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983408    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.983412    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984158    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984172    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984182    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984196    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984204    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984210    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984223    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984233    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984239    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984244    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 02 15:46:35.987683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984250    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984255    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984261    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984266    2577 flags.go:64] FLAG: --cgroup-root=""
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984270    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984275    2577 flags.go:64] FLAG: --client-ca-file=""
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984280    2577 flags.go:64] FLAG: --cloud-config=""
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984286    2577 flags.go:64] FLAG: --cloud-provider="external"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984290    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984297    2577 flags.go:64] FLAG: --cluster-domain=""
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984303    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984308    2577 flags.go:64] FLAG: --config-dir=""
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984313    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984319    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984324    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984330    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984335    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984340    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984345    2577 flags.go:64] FLAG: --contention-profiling="false"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984349    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984354    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984360    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984365    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984372    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984377    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 02 15:46:35.988204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984382    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984405    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984412    2577 flags.go:64] FLAG: --enable-server="true"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984417    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984423    2577 flags.go:64] FLAG: --event-burst="100"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984428    2577 flags.go:64] FLAG: --event-qps="50"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984433    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984438    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984443    2577 flags.go:64] FLAG: --eviction-hard=""
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984450    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984455    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984460    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984465    2577 flags.go:64] FLAG: --eviction-soft=""
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984470    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984475    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984480    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984485    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984489    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984494    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984499    2577 flags.go:64] FLAG: --feature-gates=""
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984505    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984510    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984515    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984520    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984526    2577 flags.go:64] FLAG: --healthz-port="10248"
Jun 02 15:46:35.988965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984530    2577 flags.go:64] FLAG: --help="false"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984535    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-133-168.ec2.internal"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984540    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984545    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984549    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984555    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984561    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984565    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984570    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984574    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984580    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984585    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984590    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984596    2577 flags.go:64] FLAG: --kube-reserved=""
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984601    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984606    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984611    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984616    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984621    2577 flags.go:64] FLAG: --lock-file=""
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984626    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984630    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984635    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984644    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 02 15:46:35.989598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984649    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984653    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984659    2577 flags.go:64] FLAG: --logging-format="text"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984663    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984669    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984673    2577 flags.go:64] FLAG: --manifest-url=""
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984678    2577 flags.go:64] FLAG: --manifest-url-header=""
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984685    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984689    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984696    2577 flags.go:64] FLAG: --max-pods="110"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984701    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984706    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984711    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984715    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984720    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984725    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984730    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984742    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984748    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984753    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984758    2577 flags.go:64] FLAG: --pod-cidr=""
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984763    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f1083dc6abd9805f79797fa11c9accacc4a38caa073c94cb99097faffce05983"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984773    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984778    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 02 15:46:35.990145 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984783    2577 flags.go:64] FLAG: --pods-per-core="0"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984788    2577 flags.go:64] FLAG: --port="10250"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984793    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984797    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b700e9e81cc52ce4"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984803    2577 flags.go:64] FLAG: --qos-reserved=""
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984808    2577 flags.go:64] FLAG: --read-only-port="10255"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984813    2577 flags.go:64] FLAG: --register-node="true"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984818    2577 flags.go:64] FLAG: --register-schedulable="true"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984823    2577 flags.go:64] FLAG: --register-with-taints=""
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984829    2577 flags.go:64] FLAG: --registry-burst="10"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984833    2577 flags.go:64] FLAG: --registry-qps="5"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984838    2577 flags.go:64] FLAG: --reserved-cpus=""
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984843    2577 flags.go:64] FLAG: --reserved-memory=""
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984849    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984853    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984858    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984863    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984868    2577 flags.go:64] FLAG: --runonce="false"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984873    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984878    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984882    2577 flags.go:64] FLAG: --seccomp-default="false"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984887    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984891    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984896    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984901    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984906    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jun 02 15:46:35.990737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984910    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984915    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984920    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984925    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984931    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984935    2577 flags.go:64] FLAG: --system-cgroups=""
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984940    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984949    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984953    2577 flags.go:64] FLAG: --tls-cert-file=""
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984958    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984970    2577 flags.go:64] FLAG: --tls-min-version=""
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984975    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984979    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984984    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984989    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.984994    2577 flags.go:64] FLAG: --v="2"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.985000    2577 flags.go:64] FLAG: --version="false"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.985006    2577 flags.go:64] FLAG: --vmodule=""
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.985013    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.985019    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985183    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985191    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985196    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985201    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:35.991460 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985206    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985210    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985214    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985219    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985224    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985228    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985232    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985236    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985240    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985244    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985248    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985255    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985260    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985264    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985268    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985272    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985276    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985280    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985286    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985291    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:35.992009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985295    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985299    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985303    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985308    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985312    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985316    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985320    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985325    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985329    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985333    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985337    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985341    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985345    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985349    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985354    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985358    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985362    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985366    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985370    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985375    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:35.992566 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985379    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985383    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985402    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985409    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985413    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985418    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985423    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985428    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985432    2577 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985436    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985442    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985447    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985451    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985458    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985464    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985469    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985474    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985477    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985481    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:35.993054 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985485    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985489    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985494    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985498    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985502    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985506    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985510    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985514    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985518    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985522    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985526    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985530    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985534    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985539    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985543    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985547    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985553    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985558    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985565    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:35.993538 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985572    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985577    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985582    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.985586    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.986302    2577 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.993335    2577 server.go:530] "Kubelet version" kubeletVersion="v1.33.12"
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.993353    2577 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993419    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993425    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993429    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993432    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993435    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993438    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993441    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993444    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993447    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:35.994012 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993450    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993452    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993455    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993457    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993460    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993463    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993465    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993468    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993471    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993473    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993476    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993478    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993481    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993483    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993486    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993488    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993491    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993494    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993496    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993498    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:35.994422 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993501    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993503    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993511    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993514    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993517    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993519    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993522    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993525    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993527    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993530    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993532    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993535    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993537    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993540    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993543    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993545    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993548    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993551    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993553    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993556    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993558    2577 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:35.994987 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993561    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993563    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993566    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993568    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993571    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993574    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993578    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993581    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993583    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993586    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993590    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993594    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993597    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993601    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993605    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993608    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993611    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993614    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:35.995519 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993617    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993620    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993623    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993626    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993628    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993631    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993634    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993636    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993639    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993641    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993644    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993646    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993649    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993651    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993654    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993656    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993659    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:35.995957 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993662    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.993667    2577 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993773    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993778    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993780    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993783    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993787    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993791    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993794    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993797    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993800    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993804    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993814    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993816    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993819    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 02 15:46:35.996368 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993822    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993824    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993827    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993830    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993832    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993835    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993838    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993840    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993843    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993845    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993848    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993850    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993853    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993855    2577 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993858    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993861    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993863    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993866    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993868    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993871    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 02 15:46:35.996751 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993874    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993876    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993878    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993881    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993884    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993886    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993889    2577 feature_gate.go:328] unrecognized feature gate: Example
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993892    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993894    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993897    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993900    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993903    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993905    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993908    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993910    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993913    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993915    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993918    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993920    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993922    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 02 15:46:35.997231 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993925    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993927    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993930    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993932    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993935    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993937    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993940    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993943    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993945    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993948    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993950    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993952    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993955    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993958    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993960    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993963    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993965    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993968    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993970    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993973    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 02 15:46:35.997742 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993976    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993978    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993981    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993984    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993986    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993989    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993991    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993994    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.993997    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.994000    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.994002    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.994005    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:35.994007    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.994012    2577 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
Jun 02 15:46:35.998209 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.994676    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 02 15:46:35.998569 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.996616    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 02 15:46:35.998569 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.997451    2577 server.go:1019] "Starting client certificate rotation"
Jun 02 15:46:35.998569 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.997547    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:35.998569 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:35.998380    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 02 15:46:36.024151 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.024129    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:36.026084 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.026065    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 02 15:46:36.041132 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.041112    2577 log.go:25] "Validated CRI v1 runtime API"
Jun 02 15:46:36.047173 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.047156    2577 log.go:25] "Validated CRI v1 image API"
Jun 02 15:46:36.048361 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.048340    2577 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 02 15:46:36.053382 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.053360    2577 fs.go:135] Filesystem UUIDs: map[52a9cb0a-38c8-4407-b65d-f7e90aef8908:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f6c200b8-fd8b-4e09-8ed1-881b0ce21e28:/dev/nvme0n1p4]
Jun 02 15:46:36.053483 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.053380    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 02 15:46:36.054778 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.054762    2577 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:36.057656 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.057551    2577 manager.go:217] Machine: {Timestamp:2026-06-02 15:46:36.05705052 +0000 UTC m=+0.399471734 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099561 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28ae73a922079ab46709265ca39cf3 SystemUUID:ec28ae73-a922-079a-b467-09265ca39cf3 BootID:889e23c4-7d11-4ae9-adcc-1c93f7309cd6 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8b:61:68:9d:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8b:61:68:9d:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:4e:ee:82:48:6c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 02 15:46:36.057656 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.057653    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 02 15:46:36.057754 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.057727    2577 manager.go:233] Version: {KernelVersion:5.14.0-570.117.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260521-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 02 15:46:36.058740 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.058719    2577 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 02 15:46:36.058874 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.058743    2577 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-168.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 02 15:46:36.058924 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.058886    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jun 02 15:46:36.058924 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.058895    2577 container_manager_linux.go:306] "Creating device plugin manager"
Jun 02 15:46:36.058924 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.058902    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:36.059605 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.059594    2577 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 02 15:46:36.060847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.060837    2577 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:36.060964 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.060955    2577 server.go:1267] "Using root directory" path="/var/lib/kubelet"
Jun 02 15:46:36.063206 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.063197    2577 kubelet.go:491] "Attempting to sync node with API server"
Jun 02 15:46:36.063241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.063210    2577 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 02 15:46:36.063241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.063223    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 02 15:46:36.063241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.063232    2577 kubelet.go:397] "Adding apiserver pod source"
Jun 02 15:46:36.063241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.063241    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 02 15:46:36.064229 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.064213    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:36.064294 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.064241    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 02 15:46:36.066989 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.066973    2577 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.12-3.rhaos4.20.git64c6a00.el9" apiVersion="v1"
Jun 02 15:46:36.068616 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.068598    2577 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 02 15:46:36.070805 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070788    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 02 15:46:36.070805 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070808    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070817    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070825    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070838    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070846    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070859    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070867    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070877    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070887    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070910    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 02 15:46:36.070923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.070919    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 02 15:46:36.071546 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.071528    2577 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 02 15:46:36.071601 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.071565    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 02 15:46:36.071601 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.071579    2577 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 02 15:46:36.071836 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.071816    2577 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-168.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 02 15:46:36.074654 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.074636    2577 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-168.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 02 15:46:36.074772 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.074685    2577 watchdog_linux.go:99] "Systemd watchdog is not enabled"
Jun 02 15:46:36.074772 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.074729    2577 server.go:1295] "Started kubelet"
Jun 02 15:46:36.074852 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.074761    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 02 15:46:36.075553 ip-10-0-133-168 systemd[1]: Started Kubernetes Kubelet.
Jun 02 15:46:36.077629 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.077565    2577 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 02 15:46:36.077722 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.077656    2577 server_v1.go:47] "podresources" method="list" useActivePods=true
Jun 02 15:46:36.078748 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.078729    2577 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 02 15:46:36.079329 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.079283    2577 server.go:317] "Adding debug handlers to kubelet server"
Jun 02 15:46:36.079564 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.079544    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h44sp"
Jun 02 15:46:36.084659 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.084636    2577 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 02 15:46:36.084958 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.084945    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:36.085373 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.085359    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 02 15:46:36.085458 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.085431    2577 volume_manager.go:295] "The desired_state_of_world populator starts"
Jun 02 15:46:36.085458 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.085443    2577 volume_manager.go:297] "Starting Kubelet Volume Manager"
Jun 02 15:46:36.085551 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.085504    2577 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
Jun 02 15:46:36.085637 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.085583    2577 reconstruct.go:97] "Volume reconstruction finished"
Jun 02 15:46:36.085637 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.085593    2577 reconciler.go:26] "Reconciler: start to sync state"
Jun 02 15:46:36.086039 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.086017    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.086232 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.086219    2577 factory.go:153] Registering CRI-O factory
Jun 02 15:46:36.086302 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.086272    2577 factory.go:223] Registration of the crio container factory successfully
Jun 02 15:46:36.086357 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.086325    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 02 15:46:36.086357 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.086335    2577 factory.go:55] Registering systemd factory
Jun 02 15:46:36.086357 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.086343    2577 factory.go:223] Registration of the systemd container factory successfully
Jun 02 15:46:36.086501 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.086360    2577 factory.go:103] Registering Raw factory
Jun 02 15:46:36.086501 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.086374    2577 manager.go:1196] Started watching for new ooms in manager
Jun 02 15:46:36.086882 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.086867    2577 manager.go:319] Starting recovery of all containers
Jun 02 15:46:36.088170 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.088149    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h44sp"
Jun 02 15:46:36.094416 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.094221    2577 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:36.094889 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.094870    2577 manager.go:324] Recovery completed
Jun 02 15:46:36.097654 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.097642    2577 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-133-168.ec2.internal\" not found" node="ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.100095 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.100084    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:36.101841 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.101827    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:36.101915 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.101857    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:36.101915 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.101872    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:36.103325 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.103295    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 02 15:46:36.103325 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.103319    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 02 15:46:36.103478 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.103340    2577 state_mem.go:36] "Initialized new in-memory state store"
Jun 02 15:46:36.105628 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.105612    2577 policy_none.go:49] "None policy: Start"
Jun 02 15:46:36.105711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.105633    2577 memory_manager.go:186] "Starting memorymanager" policy="None"
Jun 02 15:46:36.105711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.105643    2577 state_mem.go:35] "Initializing new in-memory state store"
Jun 02 15:46:36.150464 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.150447    2577 manager.go:341] "Starting Device Plugin manager"
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.150486    2577 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.150497    2577 server.go:85] "Starting device plugin registration server"
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.150736    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.150749    2577 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.150830    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.150912    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.150919    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.151514    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 02 15:46:36.159810 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.151551    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.216299 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.216253    2577 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
Jun 02 15:46:36.217543 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.217519    2577 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
Jun 02 15:46:36.217652 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.217549    2577 status_manager.go:230] "Starting to sync pod status with apiserver"
Jun 02 15:46:36.217652 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.217573    2577 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 02 15:46:36.217652 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.217580    2577 kubelet.go:2451] "Starting kubelet main sync loop"
Jun 02 15:46:36.217652 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.217622    2577 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 02 15:46:36.222016 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.221991    2577 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:36.251198 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.251127    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:36.252108 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.252093    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:36.252199 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.252121    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:36.252199 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.252133    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:36.252199 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.252159    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.261462 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.261445    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.261509 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.261467    2577 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-168.ec2.internal\": node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.286235 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.286215    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.318457 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.318423    2577 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"]
Jun 02 15:46:36.318572 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.318519    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:36.320967 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.320947    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:36.321063 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.320985    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:36.321063 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.320999    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:36.322128 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.322113    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:36.322289 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.322274    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.322341 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.322306    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:36.322929 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.322913    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:36.323020 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.322941    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:36.323020 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.322951    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:36.323020 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.322917    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:36.323020 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.322988    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:36.323020 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.323000    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:36.324089 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.324077    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.324132 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.324101    2577 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
Jun 02 15:46:36.324776 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.324762    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientMemory"
Jun 02 15:46:36.324825 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.324795    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasNoDiskPressure"
Jun 02 15:46:36.324825 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.324812    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeHasSufficientPID"
Jun 02 15:46:36.345002 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.344977    2577 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-168.ec2.internal\" not found" node="ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.349510 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.349491    2577 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-168.ec2.internal\" not found" node="ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.386258 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.386225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6af42d956b22a57f03f9f9ea3a9ee664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"6af42d956b22a57f03f9f9ea3a9ee664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.386258 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.386258    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42d956b22a57f03f9f9ea3a9ee664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"6af42d956b22a57f03f9f9ea3a9ee664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.386482 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.386283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72c56ec27649d987d63e9d06a174d926-config\") pod \"kube-apiserver-proxy-ip-10-0-133-168.ec2.internal\" (UID: \"72c56ec27649d987d63e9d06a174d926\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.386482 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.386313    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.486569 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.486537    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.486715 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.486603    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42d956b22a57f03f9f9ea3a9ee664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"6af42d956b22a57f03f9f9ea3a9ee664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.486715 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.486629    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72c56ec27649d987d63e9d06a174d926-config\") pod \"kube-apiserver-proxy-ip-10-0-133-168.ec2.internal\" (UID: \"72c56ec27649d987d63e9d06a174d926\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.486715 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.486647    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6af42d956b22a57f03f9f9ea3a9ee664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"6af42d956b22a57f03f9f9ea3a9ee664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.486715 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.486685    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6af42d956b22a57f03f9f9ea3a9ee664-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"6af42d956b22a57f03f9f9ea3a9ee664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.486851 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.486715    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72c56ec27649d987d63e9d06a174d926-config\") pod \"kube-apiserver-proxy-ip-10-0-133-168.ec2.internal\" (UID: \"72c56ec27649d987d63e9d06a174d926\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.486851 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.486750    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6af42d956b22a57f03f9f9ea3a9ee664-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal\" (UID: \"6af42d956b22a57f03f9f9ea3a9ee664\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.587183 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.587089    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.647307 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.647281    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.653350 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.653330    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.688105 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.688074    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.788511 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.788467    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.888905 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.888842    2577 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-168.ec2.internal\" not found"
Jun 02 15:46:36.949016 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.948991    2577 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:36.986457 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.986425    2577 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.997449 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.997426    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 02 15:46:36.997572 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.997553    2577 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:36.997610 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.997574    2577 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:36.997643 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.997610    2577 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 02 15:46:36.997643 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:36.997610    2577 kubelet.go:3342] "Failed creating a mirror pod" err="Post \"https://a1e0fc198eecb44fca1a4c5ee615912b-6e9b1f0f78428d4d.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.133.168:33670->3.210.101.18:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:36.997701 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:36.997648    2577 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal"
Jun 02 15:46:37.013617 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.013599    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 02 15:46:37.064311 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.064260    2577 apiserver.go:52] "Watching apiserver"
Jun 02 15:46:37.070464 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.070441    2577 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 02 15:46:37.072618 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.072595    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal","openshift-multus/multus-additional-cni-plugins-c6pv8","openshift-multus/network-metrics-daemon-mdgjl","kube-system/konnectivity-agent-bb8nn","kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal","openshift-cluster-node-tuning-operator/tuned-s6d6g","openshift-image-registry/node-ca-mjvd5","openshift-multus/multus-hq8fg","openshift-network-diagnostics/network-check-target-wp2zq","openshift-network-operator/iptables-alerter-8zpqk","openshift-ovn-kubernetes/ovnkube-node-xq8p6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88","openshift-dns/node-resolver-2zxzn"]
Jun 02 15:46:37.074843 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.074822    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.075710 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.075687    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.076691 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.076179    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:37.076691 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.076285    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:37.078463 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078326    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 02 15:46:37.078553 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078474    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 02 15:46:37.078553 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078543    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 02 15:46:37.078681 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078560    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lqc85\""
Jun 02 15:46:37.078681 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078550    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:37.078815 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078748    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 02 15:46:37.078815 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078771    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mxkrk\""
Jun 02 15:46:37.078815 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078775    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 02 15:46:37.078962 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078934    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 02 15:46:37.078962 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.078944    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 02 15:46:37.079066 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.079003    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 02 15:46:37.080074 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.080056    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.081045 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.081025    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.081281 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.081250    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 02 15:46:37.081664 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.081650    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wh6fq\""
Jun 02 15:46:37.081777 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.081755    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 02 15:46:37.081910 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.081895    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:37.081979 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.081960    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:37.082046 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.082032    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvjdv\""
Jun 02 15:46:37.082249 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.082236    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:37.082319 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.082305    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:37.082681 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.082662    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 02 15:46:37.083332 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.083091    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.083572 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.083553    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mnqvv\""
Jun 02 15:46:37.084332 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.084317    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.084956 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.084939    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 02 15:46:37.085080 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.085062    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 02 15:46:37.085156 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.085139    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hz8x6\""
Jun 02 15:46:37.085214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.085174    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 02 15:46:37.085409 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.085377    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.085494 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.085477    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 02 15:46:37.085996 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.085980    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 02 15:46:37.086436 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.086369    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.086522 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.086470    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cmx6c\""
Jun 02 15:46:37.086927 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.086914    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 02 15:46:37.087338 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.087320    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 02 15:46:37.087549 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.087534    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 02 15:46:37.087734 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.087720    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 02 15:46:37.088011 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.087998    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 02 15:46:37.088832 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.088817    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 02 15:46:37.089680 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089648    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-01 15:41:36 +0000 UTC" deadline="2027-11-27 02:14:58.548674873 +0000 UTC"
Jun 02 15:46:37.089680 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089675    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13018h28m21.459003402s"
Jun 02 15:46:37.089813 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07285198-8458-4168-854e-d4df187aec14-agent-certs\") pod \"konnectivity-agent-bb8nn\" (UID: \"07285198-8458-4168-854e-d4df187aec14\") " pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:37.089813 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-os-release\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.089813 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-cni-multus\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.089813 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-sys\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.089813 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089802    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 02 15:46:37.089813 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-socket-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089851    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-host\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089896    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f7rsj\""
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/497b35e9-18f2-40e6-939d-50cd84bf8ad4-serviceca\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqcsb\" (UniqueName: \"kubernetes.io/projected/497b35e9-18f2-40e6-939d-50cd84bf8ad4-kube-api-access-tqcsb\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-daemon-config\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-log-socket\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.089979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-cni-bin\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090003    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-run-netns\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090006    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-systemd\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml2zr\" (UniqueName: \"kubernetes.io/projected/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-kube-api-access-ml2zr\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysctl-d\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.090092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1c64c4e-eb38-4c9b-b944-719fe01bb403-cni-binary-copy\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-netns\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090085    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7nkv9\""
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090145    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-etc-kubernetes\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8457\" (UniqueName: \"kubernetes.io/projected/e1c64c4e-eb38-4c9b-b944-719fe01bb403-kube-api-access-n8457\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090179    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv4hs\" (UniqueName: \"kubernetes.io/projected/906be90d-c7ef-421a-ab64-22e1e9ef50c1-kube-api-access-xv4hs\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-var-lib-kubelet\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-node-log\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090245    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bdfe7b-9361-4645-bea3-a81b1c280953-tmp-dir\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090276    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-modprobe-d\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-run\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-device-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090362    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46bdfe7b-9361-4645-bea3-a81b1c280953-hosts-file\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-systemd-units\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090456    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-system-cni-dir\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090472    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cnibin\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.090718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/053754bb-d8a8-43c7-a4b4-3c8db21c5095-host-slash\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-system-cni-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-socket-dir-parent\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-cni-bin\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovn-node-metrics-cert\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090679    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrpnr\" (UniqueName: \"kubernetes.io/projected/d5d319cf-34d0-4439-a53b-6cb3dc173961-kube-api-access-vrpnr\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090696    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-tuned\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090755    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-systemd\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wbk\" (UniqueName: \"kubernetes.io/projected/2b7bc71d-feea-4854-98c8-fddb75f27a89-kube-api-access-j9wbk\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-kubelet\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-slash\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-var-lib-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090878    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.091509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090895    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovnkube-config\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-os-release\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090979    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysctl-conf\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.090996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-multus-certs\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-env-overrides\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b7bc71d-feea-4854-98c8-fddb75f27a89-tmp\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n56h\" (UniqueName: \"kubernetes.io/projected/053754bb-d8a8-43c7-a4b4-3c8db21c5095-kube-api-access-5n56h\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07285198-8458-4168-854e-d4df187aec14-konnectivity-ca\") pod \"konnectivity-agent-bb8nn\" (UID: \"07285198-8458-4168-854e-d4df187aec14\") " pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-cnibin\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091188    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-etc-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-cni-netd\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091232    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-k8s-cni-cncf-io\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091247    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-hostroot\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091269    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-conf-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.092123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-kubelet\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysconfig\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/053754bb-d8a8-43c7-a4b4-3c8db21c5095-iptables-alerter-script\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091363    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-ovn\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-lib-modules\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-registration-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/497b35e9-18f2-40e6-939d-50cd84bf8ad4-host\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-cni-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovnkube-script-lib\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9z4b\" (UniqueName: \"kubernetes.io/projected/46bdfe7b-9361-4645-bea3-a81b1c280953-kube-api-access-c9z4b\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091641    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhnsb\" (UniqueName: \"kubernetes.io/projected/f13e68f4-6625-437f-bb2c-974057826f5c-kube-api-access-qhnsb\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-sys-fs\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-kubernetes\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091784    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 02 15:46:37.092597 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.091799    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 02 15:46:37.110892 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.109972    2577 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 02 15:46:37.142021 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.141952    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af42d956b22a57f03f9f9ea3a9ee664.slice/crio-a773bf2ea9f284c556169cf1eb8ff7ae2fdd17e04eb20e9c154a242cfafcf215 WatchSource:0}: Error finding container a773bf2ea9f284c556169cf1eb8ff7ae2fdd17e04eb20e9c154a242cfafcf215: Status 404 returned error can't find the container with id a773bf2ea9f284c556169cf1eb8ff7ae2fdd17e04eb20e9c154a242cfafcf215
Jun 02 15:46:37.145199 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.145173    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c56ec27649d987d63e9d06a174d926.slice/crio-1cb29be91ea1bae3a247a73606f81adf1bf6565f20c12368f403eb40675e9112 WatchSource:0}: Error finding container 1cb29be91ea1bae3a247a73606f81adf1bf6565f20c12368f403eb40675e9112: Status 404 returned error can't find the container with id 1cb29be91ea1bae3a247a73606f81adf1bf6565f20c12368f403eb40675e9112
Jun 02 15:46:37.148838 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.148809    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:46:37.187064 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.187045    2577 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
Jun 02 15:46:37.192252 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192233    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/497b35e9-18f2-40e6-939d-50cd84bf8ad4-serviceca\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.192314 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192261    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tqcsb\" (UniqueName: \"kubernetes.io/projected/497b35e9-18f2-40e6-939d-50cd84bf8ad4-kube-api-access-tqcsb\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.192314 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192278    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-daemon-config\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.192314 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192301    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-log-socket\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.192472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192326    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-cni-bin\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.192472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192351    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-run-netns\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.192472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192372    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-systemd\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.192472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192414    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ml2zr\" (UniqueName: \"kubernetes.io/projected/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-kube-api-access-ml2zr\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.192646 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-systemd\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.192646 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192471    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-run-netns\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.192646 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192489    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-log-socket\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.192646 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-cni-bin\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.192646 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192589    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysctl-d\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.192646 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192624    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1c64c4e-eb38-4c9b-b944-719fe01bb403-cni-binary-copy\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.192913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192651    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-netns\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.192913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192676    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-etc-kubernetes\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.192913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192701    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n8457\" (UniqueName: \"kubernetes.io/projected/e1c64c4e-eb38-4c9b-b944-719fe01bb403-kube-api-access-n8457\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.192913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192726    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xv4hs\" (UniqueName: \"kubernetes.io/projected/906be90d-c7ef-421a-ab64-22e1e9ef50c1-kube-api-access-xv4hs\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.192913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192729    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/497b35e9-18f2-40e6-939d-50cd84bf8ad4-serviceca\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.192913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-netns\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.192913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192753    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-var-lib-kubelet\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.192913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192756    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysctl-d\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192792    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-node-log\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192803    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-etc-kubernetes\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192980    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-daemon-config\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192837    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-var-lib-kubelet\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192858    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-node-log\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.192962    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bdfe7b-9361-4645-bea3-a81b1c280953-tmp-dir\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193052    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193070    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1c64c4e-eb38-4c9b-b944-719fe01bb403-cni-binary-copy\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193081    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-modprobe-d\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193104    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-run\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193122    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-device-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193137    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46bdfe7b-9361-4645-bea3-a81b1c280953-hosts-file\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193170    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-systemd-units\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193185    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-modprobe-d\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/46bdfe7b-9361-4645-bea3-a81b1c280953-hosts-file\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193209    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-system-cni-dir\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.193215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193219    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-systemd-units\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193231    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-device-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193239    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-run\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193246    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cnibin\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193258    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-system-cni-dir\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193246    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46bdfe7b-9361-4645-bea3-a81b1c280953-tmp-dir\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193280    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cnibin\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193302    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/053754bb-d8a8-43c7-a4b4-3c8db21c5095-host-slash\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193327    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193360    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/053754bb-d8a8-43c7-a4b4-3c8db21c5095-host-slash\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193369    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-system-cni-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193409    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-socket-dir-parent\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193427    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-cni-bin\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-system-cni-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193442    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovn-node-metrics-cert\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193460    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vrpnr\" (UniqueName: \"kubernetes.io/projected/d5d319cf-34d0-4439-a53b-6cb3dc173961-kube-api-access-vrpnr\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.193480    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193489    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-socket-dir-parent\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.193952 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193497    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-cni-bin\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193483    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-tuned\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.193565    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs podName:f13e68f4-6625-437f-bb2c-974057826f5c nodeName:}" failed. No retries permitted until 2026-06-02 15:46:37.693528105 +0000 UTC m=+2.035949325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs") pod "network-metrics-daemon-mdgjl" (UID: "f13e68f4-6625-437f-bb2c-974057826f5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193598    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193617    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193634    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-systemd\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193650    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wbk\" (UniqueName: \"kubernetes.io/projected/2b7bc71d-feea-4854-98c8-fddb75f27a89-kube-api-access-j9wbk\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193697    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-systemd\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193719    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-kubelet\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193737    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-slash\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193762    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-var-lib-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193761    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-kubelet\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193793    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-slash\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-var-lib-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193849    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193872    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovnkube-config\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.194582 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193911    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-os-release\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193920    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193938    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.193981    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/906be90d-c7ef-421a-ab64-22e1e9ef50c1-os-release\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194170    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysctl-conf\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194187    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194217    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194458    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovnkube-config\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194462    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/906be90d-c7ef-421a-ab64-22e1e9ef50c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194502    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysctl-conf\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194536    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-multus-certs\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194564    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194591    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-env-overrides\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194640    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b7bc71d-feea-4854-98c8-fddb75f27a89-tmp\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194645    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-multus-certs\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194674    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5n56h\" (UniqueName: \"kubernetes.io/projected/053754bb-d8a8-43c7-a4b4-3c8db21c5095-kube-api-access-5n56h\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194700    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07285198-8458-4168-854e-d4df187aec14-konnectivity-ca\") pod \"konnectivity-agent-bb8nn\" (UID: \"07285198-8458-4168-854e-d4df187aec14\") " pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:37.195288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194737    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-cnibin\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194766    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-etc-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194788    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-cni-netd\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194799    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194811    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-k8s-cni-cncf-io\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194833    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-hostroot\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194853    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-conf-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194872    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-kubelet\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194892    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysconfig\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194914    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/053754bb-d8a8-43c7-a4b4-3c8db21c5095-iptables-alerter-script\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194937    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-ovn\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194961    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194975    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-run-k8s-cni-cncf-io\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194977    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-env-overrides\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.194987    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-lib-modules\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195036    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-cnibin\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195083    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-etc-openvswitch\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195111    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-conf-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-sysconfig\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195144    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-kubelet\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195159    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195180    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-cni-netd\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195185    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-run-ovn\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195187    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-registration-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195216    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/497b35e9-18f2-40e6-939d-50cd84bf8ad4-host\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195241    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-cni-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195268    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195293    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovnkube-script-lib\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195320    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c9z4b\" (UniqueName: \"kubernetes.io/projected/46bdfe7b-9361-4645-bea3-a81b1c280953-kube-api-access-c9z4b\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195377    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qhnsb\" (UniqueName: \"kubernetes.io/projected/f13e68f4-6625-437f-bb2c-974057826f5c-kube-api-access-qhnsb\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195687    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-sys-fs\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195718    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-kubernetes\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195470    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-hostroot\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195759    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07285198-8458-4168-854e-d4df187aec14-agent-certs\") pod \"konnectivity-agent-bb8nn\" (UID: \"07285198-8458-4168-854e-d4df187aec14\") " pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195777    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-kubernetes\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.196928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195581    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-lib-modules\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195789    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-os-release\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195610    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/497b35e9-18f2-40e6-939d-50cd84bf8ad4-host\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195440    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-registration-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195409    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-multus-cni-dir\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195501    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-host-run-ovn-kubernetes\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195823    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-cni-multus\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195844    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovnkube-script-lib\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195601    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/053754bb-d8a8-43c7-a4b4-3c8db21c5095-iptables-alerter-script\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195502    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195878    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-os-release\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195852    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-sys\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195916    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1c64c4e-eb38-4c9b-b944-719fe01bb403-host-var-lib-cni-multus\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195919    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-sys\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195925    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-socket-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195955    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195978    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-host\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.196014    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-etc-selinux\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.197534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.196019    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-socket-dir\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.198008 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.196042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2b7bc71d-feea-4854-98c8-fddb75f27a89-host\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.198008 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.195979    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d5d319cf-34d0-4439-a53b-6cb3dc173961-sys-fs\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.198008 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.196378    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/07285198-8458-4168-854e-d4df187aec14-konnectivity-ca\") pod \"konnectivity-agent-bb8nn\" (UID: \"07285198-8458-4168-854e-d4df187aec14\") " pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:37.198008 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.197027    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-ovn-node-metrics-cert\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.198008 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.197090    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b7bc71d-feea-4854-98c8-fddb75f27a89-tmp\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.198008 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.197127    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2b7bc71d-feea-4854-98c8-fddb75f27a89-etc-tuned\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.198171 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.198055    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/07285198-8458-4168-854e-d4df187aec14-agent-certs\") pod \"konnectivity-agent-bb8nn\" (UID: \"07285198-8458-4168-854e-d4df187aec14\") " pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:37.208061 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.208042    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqcsb\" (UniqueName: \"kubernetes.io/projected/497b35e9-18f2-40e6-939d-50cd84bf8ad4-kube-api-access-tqcsb\") pod \"node-ca-mjvd5\" (UID: \"497b35e9-18f2-40e6-939d-50cd84bf8ad4\") " pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.208962 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.208945    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:37.209047 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.208966    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:37.209047 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.208976    2577 projected.go:194] Error preparing data for projected volume kube-api-access-sbfgm for pod openshift-network-diagnostics/network-check-target-wp2zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:37.209545 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.209343    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm podName:91fef626-9a99-4972-8f0f-59ee18f994ec nodeName:}" failed. No retries permitted until 2026-06-02 15:46:37.709316394 +0000 UTC m=+2.051737607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sbfgm" (UniqueName: "kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm") pod "network-check-target-wp2zq" (UID: "91fef626-9a99-4972-8f0f-59ee18f994ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:37.210451 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.210409    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n56h\" (UniqueName: \"kubernetes.io/projected/053754bb-d8a8-43c7-a4b4-3c8db21c5095-kube-api-access-5n56h\") pod \"iptables-alerter-8zpqk\" (UID: \"053754bb-d8a8-43c7-a4b4-3c8db21c5095\") " pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.210451 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.210438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wbk\" (UniqueName: \"kubernetes.io/projected/2b7bc71d-feea-4854-98c8-fddb75f27a89-kube-api-access-j9wbk\") pod \"tuned-s6d6g\" (UID: \"2b7bc71d-feea-4854-98c8-fddb75f27a89\") " pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.213349 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.212941    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8457\" (UniqueName: \"kubernetes.io/projected/e1c64c4e-eb38-4c9b-b944-719fe01bb403-kube-api-access-n8457\") pod \"multus-hq8fg\" (UID: \"e1c64c4e-eb38-4c9b-b944-719fe01bb403\") " pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.213472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.213413    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9z4b\" (UniqueName: \"kubernetes.io/projected/46bdfe7b-9361-4645-bea3-a81b1c280953-kube-api-access-c9z4b\") pod \"node-resolver-2zxzn\" (UID: \"46bdfe7b-9361-4645-bea3-a81b1c280953\") " pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.214146 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.214124    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml2zr\" (UniqueName: \"kubernetes.io/projected/e15ec370-ab69-4d86-9aa5-23bbe054cfc0-kube-api-access-ml2zr\") pod \"ovnkube-node-xq8p6\" (UID: \"e15ec370-ab69-4d86-9aa5-23bbe054cfc0\") " pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.214146 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.214146    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhnsb\" (UniqueName: \"kubernetes.io/projected/f13e68f4-6625-437f-bb2c-974057826f5c-kube-api-access-qhnsb\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:37.214751 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.214736    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv4hs\" (UniqueName: \"kubernetes.io/projected/906be90d-c7ef-421a-ab64-22e1e9ef50c1-kube-api-access-xv4hs\") pod \"multus-additional-cni-plugins-c6pv8\" (UID: \"906be90d-c7ef-421a-ab64-22e1e9ef50c1\") " pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.214900 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.214885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrpnr\" (UniqueName: \"kubernetes.io/projected/d5d319cf-34d0-4439-a53b-6cb3dc173961-kube-api-access-vrpnr\") pod \"aws-ebs-csi-driver-node-xqg88\" (UID: \"d5d319cf-34d0-4439-a53b-6cb3dc173961\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.221492 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.221364    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal" event={"ID":"72c56ec27649d987d63e9d06a174d926","Type":"ContainerStarted","Data":"1cb29be91ea1bae3a247a73606f81adf1bf6565f20c12368f403eb40675e9112"}
Jun 02 15:46:37.222303 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.222282    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal" event={"ID":"6af42d956b22a57f03f9f9ea3a9ee664","Type":"ContainerStarted","Data":"a773bf2ea9f284c556169cf1eb8ff7ae2fdd17e04eb20e9c154a242cfafcf215"}
Jun 02 15:46:37.224937 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.224920    2577 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:37.251837 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.251808    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z4kv5"
Jun 02 15:46:37.262457 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.262437    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z4kv5"
Jun 02 15:46:37.405937 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.405851    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mjvd5"
Jun 02 15:46:37.412307 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.412280    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497b35e9_18f2_40e6_939d_50cd84bf8ad4.slice/crio-79449c535d9b48375353a66a07b50fdb7fb48767274cf08f74dfb2adf7fbc581 WatchSource:0}: Error finding container 79449c535d9b48375353a66a07b50fdb7fb48767274cf08f74dfb2adf7fbc581: Status 404 returned error can't find the container with id 79449c535d9b48375353a66a07b50fdb7fb48767274cf08f74dfb2adf7fbc581
Jun 02 15:46:37.419273 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.419255    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c6pv8"
Jun 02 15:46:37.424959 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.424937    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906be90d_c7ef_421a_ab64_22e1e9ef50c1.slice/crio-18f1530febca1be938c5ca102602cb0adb7261d84613a0746adf1a8db8327cda WatchSource:0}: Error finding container 18f1530febca1be938c5ca102602cb0adb7261d84613a0746adf1a8db8327cda: Status 404 returned error can't find the container with id 18f1530febca1be938c5ca102602cb0adb7261d84613a0746adf1a8db8327cda
Jun 02 15:46:37.425898 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.425886    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:37.432050 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.432028    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07285198_8458_4168_854e_d4df187aec14.slice/crio-94dc679edf8578ddbfad8d88d3f6c5477f5d35ea3a17ed4699ca805ec3eda6ce WatchSource:0}: Error finding container 94dc679edf8578ddbfad8d88d3f6c5477f5d35ea3a17ed4699ca805ec3eda6ce: Status 404 returned error can't find the container with id 94dc679edf8578ddbfad8d88d3f6c5477f5d35ea3a17ed4699ca805ec3eda6ce
Jun 02 15:46:37.446513 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.446494    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s6d6g"
Jun 02 15:46:37.452165 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.452141    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hq8fg"
Jun 02 15:46:37.452255 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.452235    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7bc71d_feea_4854_98c8_fddb75f27a89.slice/crio-40dca53dd4ab175afb50972929a1ab4c5f7c745aaeccac51d5e0a94aa0313847 WatchSource:0}: Error finding container 40dca53dd4ab175afb50972929a1ab4c5f7c745aaeccac51d5e0a94aa0313847: Status 404 returned error can't find the container with id 40dca53dd4ab175afb50972929a1ab4c5f7c745aaeccac51d5e0a94aa0313847
Jun 02 15:46:37.458110 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.458088    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8zpqk"
Jun 02 15:46:37.458355 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.458331    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c64c4e_eb38_4c9b_b944_719fe01bb403.slice/crio-a1bffcb6b8f888b3c92b7b4fd61dd8c77c2a8efc882bfd4a536a794f06179986 WatchSource:0}: Error finding container a1bffcb6b8f888b3c92b7b4fd61dd8c77c2a8efc882bfd4a536a794f06179986: Status 404 returned error can't find the container with id a1bffcb6b8f888b3c92b7b4fd61dd8c77c2a8efc882bfd4a536a794f06179986
Jun 02 15:46:37.463934 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.463915    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:46:37.466309 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.466290    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053754bb_d8a8_43c7_a4b4_3c8db21c5095.slice/crio-8378ce2798fc67a5d4be6d64dc37067b6e6cb27458880c4c075be877fbd10959 WatchSource:0}: Error finding container 8378ce2798fc67a5d4be6d64dc37067b6e6cb27458880c4c075be877fbd10959: Status 404 returned error can't find the container with id 8378ce2798fc67a5d4be6d64dc37067b6e6cb27458880c4c075be877fbd10959
Jun 02 15:46:37.469851 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.469833    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88"
Jun 02 15:46:37.471409 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.471367    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15ec370_ab69_4d86_9aa5_23bbe054cfc0.slice/crio-345fa724afcca55783e9c9220c98ad27955a339be7e654a41f14fdafb2dcbb13 WatchSource:0}: Error finding container 345fa724afcca55783e9c9220c98ad27955a339be7e654a41f14fdafb2dcbb13: Status 404 returned error can't find the container with id 345fa724afcca55783e9c9220c98ad27955a339be7e654a41f14fdafb2dcbb13
Jun 02 15:46:37.473472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.473433    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2zxzn"
Jun 02 15:46:37.479095 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.478516    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d319cf_34d0_4439_a53b_6cb3dc173961.slice/crio-dbaa9e26ab0f3d88a2ae177dcf3938cca868baa9fe54b1b97a33a58638d5debf WatchSource:0}: Error finding container dbaa9e26ab0f3d88a2ae177dcf3938cca868baa9fe54b1b97a33a58638d5debf: Status 404 returned error can't find the container with id dbaa9e26ab0f3d88a2ae177dcf3938cca868baa9fe54b1b97a33a58638d5debf
Jun 02 15:46:37.482171 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:46:37.482149    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bdfe7b_9361_4645_bea3_a81b1c280953.slice/crio-fe1e9202fd8727996801899836eb1ae67bd58e3a9c50750efe4db8465cada6eb WatchSource:0}: Error finding container fe1e9202fd8727996801899836eb1ae67bd58e3a9c50750efe4db8465cada6eb: Status 404 returned error can't find the container with id fe1e9202fd8727996801899836eb1ae67bd58e3a9c50750efe4db8465cada6eb
Jun 02 15:46:37.698501 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.698419    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:37.698659 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.698588    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:37.698729 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.698705    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs podName:f13e68f4-6625-437f-bb2c-974057826f5c nodeName:}" failed. No retries permitted until 2026-06-02 15:46:38.698683164 +0000 UTC m=+3.041104388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs") pod "network-metrics-daemon-mdgjl" (UID: "f13e68f4-6625-437f-bb2c-974057826f5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:37.798851 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:37.798804    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:37.799050 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.799032    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:37.799152 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.799061    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:37.799152 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.799079    2577 projected.go:194] Error preparing data for projected volume kube-api-access-sbfgm for pod openshift-network-diagnostics/network-check-target-wp2zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:37.799304 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:37.799151    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm podName:91fef626-9a99-4972-8f0f-59ee18f994ec nodeName:}" failed. No retries permitted until 2026-06-02 15:46:38.799130509 +0000 UTC m=+3.141551726 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbfgm" (UniqueName: "kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm") pod "network-check-target-wp2zq" (UID: "91fef626-9a99-4972-8f0f-59ee18f994ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:38.157001 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.156637    2577 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:38.200403 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.200183    2577 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:38.219683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.219078    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:38.219683 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:38.219257    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:38.219683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.219274    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:38.219683 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:38.219403    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:38.238547 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.238351    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88" event={"ID":"d5d319cf-34d0-4439-a53b-6cb3dc173961","Type":"ContainerStarted","Data":"dbaa9e26ab0f3d88a2ae177dcf3938cca868baa9fe54b1b97a33a58638d5debf"}
Jun 02 15:46:38.242866 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.242789    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bb8nn" event={"ID":"07285198-8458-4168-854e-d4df187aec14","Type":"ContainerStarted","Data":"94dc679edf8578ddbfad8d88d3f6c5477f5d35ea3a17ed4699ca805ec3eda6ce"}
Jun 02 15:46:38.245309 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.245281    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" event={"ID":"906be90d-c7ef-421a-ab64-22e1e9ef50c1","Type":"ContainerStarted","Data":"18f1530febca1be938c5ca102602cb0adb7261d84613a0746adf1a8db8327cda"}
Jun 02 15:46:38.249237 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.249180    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mjvd5" event={"ID":"497b35e9-18f2-40e6-939d-50cd84bf8ad4","Type":"ContainerStarted","Data":"79449c535d9b48375353a66a07b50fdb7fb48767274cf08f74dfb2adf7fbc581"}
Jun 02 15:46:38.252600 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.252572    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2zxzn" event={"ID":"46bdfe7b-9361-4645-bea3-a81b1c280953","Type":"ContainerStarted","Data":"fe1e9202fd8727996801899836eb1ae67bd58e3a9c50750efe4db8465cada6eb"}
Jun 02 15:46:38.263425 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.263339    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:37 +0000 UTC" deadline="2028-02-01 12:00:42.902874383 +0000 UTC"
Jun 02 15:46:38.263425 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.263373    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14612h14m4.639505633s"
Jun 02 15:46:38.267061 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.266746    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"345fa724afcca55783e9c9220c98ad27955a339be7e654a41f14fdafb2dcbb13"}
Jun 02 15:46:38.274947 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.274905    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8zpqk" event={"ID":"053754bb-d8a8-43c7-a4b4-3c8db21c5095","Type":"ContainerStarted","Data":"8378ce2798fc67a5d4be6d64dc37067b6e6cb27458880c4c075be877fbd10959"}
Jun 02 15:46:38.299414 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.299326    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hq8fg" event={"ID":"e1c64c4e-eb38-4c9b-b944-719fe01bb403","Type":"ContainerStarted","Data":"a1bffcb6b8f888b3c92b7b4fd61dd8c77c2a8efc882bfd4a536a794f06179986"}
Jun 02 15:46:38.322642 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.322604    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s6d6g" event={"ID":"2b7bc71d-feea-4854-98c8-fddb75f27a89","Type":"ContainerStarted","Data":"40dca53dd4ab175afb50972929a1ab4c5f7c745aaeccac51d5e0a94aa0313847"}
Jun 02 15:46:38.635282 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.635166    2577 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 02 15:46:38.707944 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.707910    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:38.708204 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:38.708088    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:38.708204 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:38.708175    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs podName:f13e68f4-6625-437f-bb2c-974057826f5c nodeName:}" failed. No retries permitted until 2026-06-02 15:46:40.708153191 +0000 UTC m=+5.050574416 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs") pod "network-metrics-daemon-mdgjl" (UID: "f13e68f4-6625-437f-bb2c-974057826f5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:38.808983 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:38.808947    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:38.809190 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:38.809152    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:38.809190 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:38.809172    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:38.809190 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:38.809185    2577 projected.go:194] Error preparing data for projected volume kube-api-access-sbfgm for pod openshift-network-diagnostics/network-check-target-wp2zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:38.809359 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:38.809243    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm podName:91fef626-9a99-4972-8f0f-59ee18f994ec nodeName:}" failed. No retries permitted until 2026-06-02 15:46:40.809226055 +0000 UTC m=+5.151647259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbfgm" (UniqueName: "kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm") pod "network-check-target-wp2zq" (UID: "91fef626-9a99-4972-8f0f-59ee18f994ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:39.264314 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:39.264266    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-01 15:41:37 +0000 UTC" deadline="2028-02-17 11:51:28.043671 +0000 UTC"
Jun 02 15:46:39.264314 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:39.264311    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14996h4m48.779363731s"
Jun 02 15:46:40.222506 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:40.222476    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:40.222740 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:40.222616    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:40.223193 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:40.222992    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:40.223193 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:40.223084    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:40.725717 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:40.725675    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:40.726200 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:40.725828    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:40.726200 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:40.725891    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs podName:f13e68f4-6625-437f-bb2c-974057826f5c nodeName:}" failed. No retries permitted until 2026-06-02 15:46:44.725873091 +0000 UTC m=+9.068294294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs") pod "network-metrics-daemon-mdgjl" (UID: "f13e68f4-6625-437f-bb2c-974057826f5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:40.827187 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:40.826574    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:40.827187 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:40.826735    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:40.827187 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:40.826753    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:40.827187 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:40.826766    2577 projected.go:194] Error preparing data for projected volume kube-api-access-sbfgm for pod openshift-network-diagnostics/network-check-target-wp2zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:40.827187 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:40.826827    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm podName:91fef626-9a99-4972-8f0f-59ee18f994ec nodeName:}" failed. No retries permitted until 2026-06-02 15:46:44.826807306 +0000 UTC m=+9.169228512 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbfgm" (UniqueName: "kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm") pod "network-check-target-wp2zq" (UID: "91fef626-9a99-4972-8f0f-59ee18f994ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:42.218119 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:42.218084    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:42.218119 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:42.218124    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:42.218659 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:42.218218    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:42.218659 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:42.218327    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:44.218673 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:44.218640    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:44.219090 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:44.218785    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:44.219159 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:44.218643    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:44.219270 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:44.219246    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:44.758376 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:44.758340    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:44.758578 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:44.758528    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:44.758578 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:44.758577    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs podName:f13e68f4-6625-437f-bb2c-974057826f5c nodeName:}" failed. No retries permitted until 2026-06-02 15:46:52.758564054 +0000 UTC m=+17.100985253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs") pod "network-metrics-daemon-mdgjl" (UID: "f13e68f4-6625-437f-bb2c-974057826f5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:44.858892 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:44.858761    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:44.859085 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:44.858932    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:44.859085 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:44.858952    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:44.859085 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:44.858963    2577 projected.go:194] Error preparing data for projected volume kube-api-access-sbfgm for pod openshift-network-diagnostics/network-check-target-wp2zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:44.859085 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:44.859023    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm podName:91fef626-9a99-4972-8f0f-59ee18f994ec nodeName:}" failed. No retries permitted until 2026-06-02 15:46:52.859004411 +0000 UTC m=+17.201425625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbfgm" (UniqueName: "kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm") pod "network-check-target-wp2zq" (UID: "91fef626-9a99-4972-8f0f-59ee18f994ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:46.220857 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:46.220824    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:46.221292 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:46.220824    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:46.221292 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:46.220950    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:46.221292 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:46.220998    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:48.218243 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:48.218029    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:48.218723 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:48.218036    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:48.218723 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:48.218454    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:48.218723 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:48.218330    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:50.218151 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:50.218114    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:50.218608 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:50.218263    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:50.218608 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:50.218321    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:50.218608 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:50.218448    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:52.217931 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:52.217894    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:52.218318 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:52.217901    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:52.218318 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:52.217997    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:52.218318 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:52.218078    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:52.811602 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:52.811521    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:52.811794 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:52.811660    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:52.811794 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:52.811741    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs podName:f13e68f4-6625-437f-bb2c-974057826f5c nodeName:}" failed. No retries permitted until 2026-06-02 15:47:08.811722124 +0000 UTC m=+33.154143326 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs") pod "network-metrics-daemon-mdgjl" (UID: "f13e68f4-6625-437f-bb2c-974057826f5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:46:52.912514 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:52.912476    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:52.912678 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:52.912654    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:46:52.912755 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:52.912679    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:46:52.912755 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:52.912692    2577 projected.go:194] Error preparing data for projected volume kube-api-access-sbfgm for pod openshift-network-diagnostics/network-check-target-wp2zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:52.912891 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:52.912764    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm podName:91fef626-9a99-4972-8f0f-59ee18f994ec nodeName:}" failed. No retries permitted until 2026-06-02 15:47:08.912743161 +0000 UTC m=+33.255164378 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbfgm" (UniqueName: "kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm") pod "network-check-target-wp2zq" (UID: "91fef626-9a99-4972-8f0f-59ee18f994ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:46:54.218596 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:54.218561    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:54.219052 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:54.218739    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:54.219052 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:54.218798    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:54.219052 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:54.218904    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:56.220120 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:56.220050    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:56.220479 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:56.220059    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:56.220479 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:56.220220    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:56.220479 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:56.220139    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:57.359871 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.359648    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"e957d3a405d1c90402d2446d7a49f25aa32951b92ab82551417134b72fba51c8"}
Jun 02 15:46:57.360445 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.359893    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"721769a5abd19d5f96e5b7aa2f4116e86d041b2383019440a6213cef6f5853d3"}
Jun 02 15:46:57.360445 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.359918    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"cc5ca736b5f5339e178fc6d38032c4904afe48491b65f2ebf3347fbb1853e816"}
Jun 02 15:46:57.360445 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.359932    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"5f9154552bc12a7533226ff51945d1c9cbe53a16217c2b01c12bdbe94f9f39ac"}
Jun 02 15:46:57.360445 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.359942    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"036f810d59763c75aa5963dfefc003627fc3edbb08d0d8a48dbd05a47b848d8a"}
Jun 02 15:46:57.360445 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.359950    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"55341186cbff4d7a47da9767d282893ae0cf2f05515182c192271f564e4c43c6"}
Jun 02 15:46:57.361120 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.361093    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hq8fg" event={"ID":"e1c64c4e-eb38-4c9b-b944-719fe01bb403","Type":"ContainerStarted","Data":"af1d69c8013df20dee9abd46853cd0d45ea3c3cc44c0407be406a96197ed9dc3"}
Jun 02 15:46:57.362427 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.362383    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s6d6g" event={"ID":"2b7bc71d-feea-4854-98c8-fddb75f27a89","Type":"ContainerStarted","Data":"d46979ec90d4ac8223abab5d403113d5a667ee1ff0aec653f68209896ee7a5f7"}
Jun 02 15:46:57.363828 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.363802    2577 generic.go:358] "Generic (PLEG): container finished" podID="6af42d956b22a57f03f9f9ea3a9ee664" containerID="3da8564d848d3e6556833f787a4a6b5d56fa255779241a023c0d95277c1ad866" exitCode=0
Jun 02 15:46:57.363913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.363883    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal" event={"ID":"6af42d956b22a57f03f9f9ea3a9ee664","Type":"ContainerDied","Data":"3da8564d848d3e6556833f787a4a6b5d56fa255779241a023c0d95277c1ad866"}
Jun 02 15:46:57.365442 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.365424    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88" event={"ID":"d5d319cf-34d0-4439-a53b-6cb3dc173961","Type":"ContainerStarted","Data":"175a19839adf90ef088b5ced891bcac4ebf08530c87bbb1f2ed291a44ead1ebd"}
Jun 02 15:46:57.366782 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.366759    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bb8nn" event={"ID":"07285198-8458-4168-854e-d4df187aec14","Type":"ContainerStarted","Data":"9c51c3c97995ef7bd97e1b19a6162f935447a8851e781b25072648c7db0eed12"}
Jun 02 15:46:57.368126 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.368098    2577 generic.go:358] "Generic (PLEG): container finished" podID="906be90d-c7ef-421a-ab64-22e1e9ef50c1" containerID="1333ab85db8024f9e8b1f2f392ee9009db1dad7daff9fa1a988b45f0866c6927" exitCode=0
Jun 02 15:46:57.368220 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.368158    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" event={"ID":"906be90d-c7ef-421a-ab64-22e1e9ef50c1","Type":"ContainerDied","Data":"1333ab85db8024f9e8b1f2f392ee9009db1dad7daff9fa1a988b45f0866c6927"}
Jun 02 15:46:57.369545 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.369508    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mjvd5" event={"ID":"497b35e9-18f2-40e6-939d-50cd84bf8ad4","Type":"ContainerStarted","Data":"9be421b1b5f403cac4bc461db4e31f2a48de54ffa2d4fb66f7c5d570fd0b6a02"}
Jun 02 15:46:57.371336 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.371035    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal" event={"ID":"72c56ec27649d987d63e9d06a174d926","Type":"ContainerStarted","Data":"7ddea8ce9eaca5aa981124d7137af1211aba3f8f0795e8911a89b8c177cb5593"}
Jun 02 15:46:57.373849 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.373798    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2zxzn" event={"ID":"46bdfe7b-9361-4645-bea3-a81b1c280953","Type":"ContainerStarted","Data":"2c760ec886d402611beeb839bd91d2fe3e8d8584cb51aff79a9862dcee4abce6"}
Jun 02 15:46:57.378156 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.378115    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-hq8fg" podStartSLOduration=2.341876624 podStartE2EDuration="21.378104973s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.462516102 +0000 UTC m=+1.804937301" lastFinishedPulling="2026-06-02 15:46:56.498744446 +0000 UTC m=+20.841165650" observedRunningTime="2026-06-02 15:46:57.377950438 +0000 UTC m=+21.720371660" watchObservedRunningTime="2026-06-02 15:46:57.378104973 +0000 UTC m=+21.720526195"
Jun 02 15:46:57.393611 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.393278    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mjvd5" podStartSLOduration=2.872528496 podStartE2EDuration="21.393264417s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.413880779 +0000 UTC m=+1.756301982" lastFinishedPulling="2026-06-02 15:46:55.934616703 +0000 UTC m=+20.277037903" observedRunningTime="2026-06-02 15:46:57.392742439 +0000 UTC m=+21.735163662" watchObservedRunningTime="2026-06-02 15:46:57.393264417 +0000 UTC m=+21.735685669"
Jun 02 15:46:57.408509 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.408448    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-2zxzn" podStartSLOduration=2.435336871 podStartE2EDuration="21.40842885s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.483952973 +0000 UTC m=+1.826374175" lastFinishedPulling="2026-06-02 15:46:56.457044944 +0000 UTC m=+20.799466154" observedRunningTime="2026-06-02 15:46:57.407865118 +0000 UTC m=+21.750286331" watchObservedRunningTime="2026-06-02 15:46:57.40842885 +0000 UTC m=+21.750850072"
Jun 02 15:46:57.440660 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.440611    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-168.ec2.internal" podStartSLOduration=20.440593852 podStartE2EDuration="20.440593852s" podCreationTimestamp="2026-06-02 15:46:37 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:57.440497511 +0000 UTC m=+21.782918731" watchObservedRunningTime="2026-06-02 15:46:57.440593852 +0000 UTC m=+21.783015076"
Jun 02 15:46:57.453893 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.453863    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:57.454507 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.454488    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:57.487498 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.487460    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bb8nn" podStartSLOduration=2.581659275 podStartE2EDuration="21.487445946s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.433275231 +0000 UTC m=+1.775696431" lastFinishedPulling="2026-06-02 15:46:56.339061902 +0000 UTC m=+20.681483102" observedRunningTime="2026-06-02 15:46:57.487031256 +0000 UTC m=+21.829452479" watchObservedRunningTime="2026-06-02 15:46:57.487445946 +0000 UTC m=+21.829867168"
Jun 02 15:46:57.507073 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.507046    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:57.507503 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.507484    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bb8nn"
Jun 02 15:46:57.531189 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:57.531061    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s6d6g" podStartSLOduration=2.527735122 podStartE2EDuration="21.531045926s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.453826758 +0000 UTC m=+1.796247958" lastFinishedPulling="2026-06-02 15:46:56.457137561 +0000 UTC m=+20.799558762" observedRunningTime="2026-06-02 15:46:57.513674711 +0000 UTC m=+21.856095934" watchObservedRunningTime="2026-06-02 15:46:57.531045926 +0000 UTC m=+21.873467148"
Jun 02 15:46:58.040165 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.040142    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 02 15:46:58.162460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.162330    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-02T15:46:58.040161151Z","UUID":"294d526f-a60c-448f-adf3-678abb46617e","Handler":null,"Name":"","Endpoint":""}
Jun 02 15:46:58.165895 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.165868    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 02 15:46:58.165895 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.165899    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 02 15:46:58.218624 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.218549    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:46:58.218783 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.218550    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:46:58.218783 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:58.218675    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:46:58.218783 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:46:58.218733    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:46:58.376579 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.376535    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8zpqk" event={"ID":"053754bb-d8a8-43c7-a4b4-3c8db21c5095","Type":"ContainerStarted","Data":"1540a3e8bd6330d045e1e4ed22c3c1658e1da7c1836a054dd7cf5488ddbc98d4"}
Jun 02 15:46:58.378398 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.378355    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal" event={"ID":"6af42d956b22a57f03f9f9ea3a9ee664","Type":"ContainerStarted","Data":"e5bf1027500141787d14e92258b78a1576af8c4dfc25e00531eb6654ca29723d"}
Jun 02 15:46:58.380052 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.380025    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88" event={"ID":"d5d319cf-34d0-4439-a53b-6cb3dc173961","Type":"ContainerStarted","Data":"a61d5ae7f5cabe5051f51aa6f5ca93942c3698ead05a731ea5891b77fcaeff24"}
Jun 02 15:46:58.392879 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.392832    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8zpqk" podStartSLOduration=3.406705689 podStartE2EDuration="22.392816835s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.468046799 +0000 UTC m=+1.810467999" lastFinishedPulling="2026-06-02 15:46:56.454157927 +0000 UTC m=+20.796579145" observedRunningTime="2026-06-02 15:46:58.392696308 +0000 UTC m=+22.735117543" watchObservedRunningTime="2026-06-02 15:46:58.392816835 +0000 UTC m=+22.735238058"
Jun 02 15:46:58.407730 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:58.407688    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-168.ec2.internal" podStartSLOduration=22.407672616 podStartE2EDuration="22.407672616s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:46:58.407230353 +0000 UTC m=+22.749651576" watchObservedRunningTime="2026-06-02 15:46:58.407672616 +0000 UTC m=+22.750093839"
Jun 02 15:46:59.384142 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:59.384101    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88" event={"ID":"d5d319cf-34d0-4439-a53b-6cb3dc173961","Type":"ContainerStarted","Data":"68a89673769b56224b552e2e57cbe27b781cd5e4756726ebe42f10ccd2144a26"}
Jun 02 15:46:59.404439 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:46:59.404370    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xqg88" podStartSLOduration=2.03928464 podStartE2EDuration="23.404356263s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.481091835 +0000 UTC m=+1.823513035" lastFinishedPulling="2026-06-02 15:46:58.846163453 +0000 UTC m=+23.188584658" observedRunningTime="2026-06-02 15:46:59.403957235 +0000 UTC m=+23.746378459" watchObservedRunningTime="2026-06-02 15:46:59.404356263 +0000 UTC m=+23.746777486"
Jun 02 15:47:00.217996 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:00.217955    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:00.218149 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:00.218049    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:00.218204 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:00.218151    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:47:00.218247 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:00.218203    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:47:00.389739 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:00.389702    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"b9c37cda881717607cee34d1864299d1926b2a2acb12b5d8827301b06ef5296e"}
Jun 02 15:47:02.217939 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.217765    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:02.218725 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.217825    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:02.218725 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:02.218299    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:47:02.218725 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:02.218604    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:47:02.394336 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.394304    2577 generic.go:358] "Generic (PLEG): container finished" podID="906be90d-c7ef-421a-ab64-22e1e9ef50c1" containerID="c914e004d2b3f1981ce2e1f3e3a97ffdb5b98f412e7977fa0ee5de8c8c169c4b" exitCode=0
Jun 02 15:47:02.394535 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.394404    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" event={"ID":"906be90d-c7ef-421a-ab64-22e1e9ef50c1","Type":"ContainerDied","Data":"c914e004d2b3f1981ce2e1f3e3a97ffdb5b98f412e7977fa0ee5de8c8c169c4b"}
Jun 02 15:47:02.397554 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.397528    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" event={"ID":"e15ec370-ab69-4d86-9aa5-23bbe054cfc0","Type":"ContainerStarted","Data":"c2cb190aea8b3174a093e1badf8cdc44e8f22c9a76384b75b883d482c06fd052"}
Jun 02 15:47:02.397794 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.397772    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:47:02.397888 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.397800    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:47:02.397888 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.397812    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:47:02.413524 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.413497    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:47:02.413673 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.413569    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:47:02.444312 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:02.444269    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6" podStartSLOduration=7.157127106 podStartE2EDuration="26.444255339s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.4731643 +0000 UTC m=+1.815585512" lastFinishedPulling="2026-06-02 15:46:56.760292546 +0000 UTC m=+21.102713745" observedRunningTime="2026-06-02 15:47:02.44420802 +0000 UTC m=+26.786629239" watchObservedRunningTime="2026-06-02 15:47:02.444255339 +0000 UTC m=+26.786676560"
Jun 02 15:47:03.401196 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:03.400927    2577 generic.go:358] "Generic (PLEG): container finished" podID="906be90d-c7ef-421a-ab64-22e1e9ef50c1" containerID="e353faf00824d4a4f01c8f566f4360822f2cfc16ca3e89ff64b4d57198c9c8a9" exitCode=0
Jun 02 15:47:03.401196 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:03.401018    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" event={"ID":"906be90d-c7ef-421a-ab64-22e1e9ef50c1","Type":"ContainerDied","Data":"e353faf00824d4a4f01c8f566f4360822f2cfc16ca3e89ff64b4d57198c9c8a9"}
Jun 02 15:47:03.470968 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:03.470936    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mdgjl"]
Jun 02 15:47:03.471123 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:03.471066    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:03.471192 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:03.471170    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:47:03.474070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:03.474045    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wp2zq"]
Jun 02 15:47:03.474195 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:03.474145    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:03.474248 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:03.474215    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:47:04.404946 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:04.404857    2577 generic.go:358] "Generic (PLEG): container finished" podID="906be90d-c7ef-421a-ab64-22e1e9ef50c1" containerID="b86ed5a76cc7861070a7c6e2255cafce14a530afeebc46c3c7f62dbbed76ee3a" exitCode=0
Jun 02 15:47:04.405367 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:04.404949    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" event={"ID":"906be90d-c7ef-421a-ab64-22e1e9ef50c1","Type":"ContainerDied","Data":"b86ed5a76cc7861070a7c6e2255cafce14a530afeebc46c3c7f62dbbed76ee3a"}
Jun 02 15:47:05.218375 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:05.218343    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:05.218533 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:05.218349    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:05.218533 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:05.218473    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:47:05.218595 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:05.218540    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:47:07.217948 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:07.217912    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:07.218378 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:07.217910    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:07.218378 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:07.218037    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:47:07.218378 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:07.218124    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:47:08.830080 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:08.829880    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:08.830526 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:08.830054    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:08.830526 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:08.830193    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs podName:f13e68f4-6625-437f-bb2c-974057826f5c nodeName:}" failed. No retries permitted until 2026-06-02 15:47:40.830172212 +0000 UTC m=+65.172593415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs") pod "network-metrics-daemon-mdgjl" (UID: "f13e68f4-6625-437f-bb2c-974057826f5c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 02 15:47:08.931128 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:08.931076    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:08.931317 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:08.931277    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 02 15:47:08.931317 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:08.931304    2577 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 02 15:47:08.931317 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:08.931318    2577 projected.go:194] Error preparing data for projected volume kube-api-access-sbfgm for pod openshift-network-diagnostics/network-check-target-wp2zq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:08.931484 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:08.931400    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm podName:91fef626-9a99-4972-8f0f-59ee18f994ec nodeName:}" failed. No retries permitted until 2026-06-02 15:47:40.931367521 +0000 UTC m=+65.273788725 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sbfgm" (UniqueName: "kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm") pod "network-check-target-wp2zq" (UID: "91fef626-9a99-4972-8f0f-59ee18f994ec") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 02 15:47:09.218677 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.218631    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:09.218867 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.218653    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:09.218867 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:09.218780    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wp2zq" podUID="91fef626-9a99-4972-8f0f-59ee18f994ec"
Jun 02 15:47:09.218980 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:09.218886    2577 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mdgjl" podUID="f13e68f4-6625-437f-bb2c-974057826f5c"
Jun 02 15:47:09.453352 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.453278    2577 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-168.ec2.internal" event="NodeReady"
Jun 02 15:47:09.453568 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.453468    2577 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
Jun 02 15:47:09.505939 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.505907    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jllqh"]
Jun 02 15:47:09.534858 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.534827    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-d6xhm"]
Jun 02 15:47:09.535011 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.534996    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.537379 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.537352    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 02 15:47:09.537526 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.537403    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 02 15:47:09.537754 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.537727    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6cp8\""
Jun 02 15:47:09.550936 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.550910    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d6xhm"]
Jun 02 15:47:09.551067 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.550942    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jllqh"]
Jun 02 15:47:09.551067 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.551057    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:09.553652 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.553319    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 02 15:47:09.553652 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.553366    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ws8nx\""
Jun 02 15:47:09.553652 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.553368    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 02 15:47:09.553652 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.553370    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 02 15:47:09.636890 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.636848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:09.637051 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.636906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dc34aa-0d48-4060-a21d-c93066f8c683-tmp-dir\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.637051 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.636935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2f27\" (UniqueName: \"kubernetes.io/projected/84dc34aa-0d48-4060-a21d-c93066f8c683-kube-api-access-d2f27\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.637051 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.636963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84dc34aa-0d48-4060-a21d-c93066f8c683-config-volume\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.637051 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.637025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9zj\" (UniqueName: \"kubernetes.io/projected/3384c3b6-30a7-4de5-86d6-a139400ecaf6-kube-api-access-kl9zj\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:09.637209 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.637058    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.738204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.738115    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.738204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.738174    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:09.738470 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:09.738270    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:09.738470 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.738290    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dc34aa-0d48-4060-a21d-c93066f8c683-tmp-dir\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.738470 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.738330    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d2f27\" (UniqueName: \"kubernetes.io/projected/84dc34aa-0d48-4060-a21d-c93066f8c683-kube-api-access-d2f27\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.738470 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:09.738278    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:09.738470 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:09.738363    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls podName:84dc34aa-0d48-4060-a21d-c93066f8c683 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:10.238342402 +0000 UTC m=+34.580763621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls") pod "dns-default-jllqh" (UID: "84dc34aa-0d48-4060-a21d-c93066f8c683") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:09.738470 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.738413    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84dc34aa-0d48-4060-a21d-c93066f8c683-config-volume\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.738470 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:09.738437    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert podName:3384c3b6-30a7-4de5-86d6-a139400ecaf6 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:10.238419714 +0000 UTC m=+34.580840928 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert") pod "ingress-canary-d6xhm" (UID: "3384c3b6-30a7-4de5-86d6-a139400ecaf6") : secret "canary-serving-cert" not found
Jun 02 15:47:09.738811 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.738509    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9zj\" (UniqueName: \"kubernetes.io/projected/3384c3b6-30a7-4de5-86d6-a139400ecaf6-kube-api-access-kl9zj\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:09.738811 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.738711    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84dc34aa-0d48-4060-a21d-c93066f8c683-tmp-dir\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.738936 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.738917    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84dc34aa-0d48-4060-a21d-c93066f8c683-config-volume\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.750091 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.750069    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2f27\" (UniqueName: \"kubernetes.io/projected/84dc34aa-0d48-4060-a21d-c93066f8c683-kube-api-access-d2f27\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:09.750268 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:09.750197    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9zj\" (UniqueName: \"kubernetes.io/projected/3384c3b6-30a7-4de5-86d6-a139400ecaf6-kube-api-access-kl9zj\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:10.242548 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:10.242510    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:10.243121 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:10.242567    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:10.243121 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:10.242669    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:10.243121 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:10.242682    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:10.243121 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:10.242738    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls podName:84dc34aa-0d48-4060-a21d-c93066f8c683 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:11.242720803 +0000 UTC m=+35.585142017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls") pod "dns-default-jllqh" (UID: "84dc34aa-0d48-4060-a21d-c93066f8c683") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:10.243121 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:10.242756    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert podName:3384c3b6-30a7-4de5-86d6-a139400ecaf6 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:11.242747994 +0000 UTC m=+35.585169194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert") pod "ingress-canary-d6xhm" (UID: "3384c3b6-30a7-4de5-86d6-a139400ecaf6") : secret "canary-serving-cert" not found
Jun 02 15:47:11.218613 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.218576    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:11.218772 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.218585    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:11.221486 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.221464    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:11.221623 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.221534    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:11.221623 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.221538    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2vmzq\""
Jun 02 15:47:11.221623 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.221590    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6qg5h\""
Jun 02 15:47:11.221623 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.221538    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:11.249473 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.249442    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:11.250062 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:11.249597    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:11.250062 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.249602    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:11.250062 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:11.249662    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls podName:84dc34aa-0d48-4060-a21d-c93066f8c683 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:13.249644621 +0000 UTC m=+37.592065835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls") pod "dns-default-jllqh" (UID: "84dc34aa-0d48-4060-a21d-c93066f8c683") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:11.250062 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:11.249707    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:11.250062 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:11.249774    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert podName:3384c3b6-30a7-4de5-86d6-a139400ecaf6 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:13.249762236 +0000 UTC m=+37.592183435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert") pod "ingress-canary-d6xhm" (UID: "3384c3b6-30a7-4de5-86d6-a139400ecaf6") : secret "canary-serving-cert" not found
Jun 02 15:47:11.420271 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.420237    2577 generic.go:358] "Generic (PLEG): container finished" podID="906be90d-c7ef-421a-ab64-22e1e9ef50c1" containerID="d0ed14b90cdea589c11fba79e9fc11afc5640cd6457d2bdd5661953101053d5e" exitCode=0
Jun 02 15:47:11.420444 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:11.420277    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" event={"ID":"906be90d-c7ef-421a-ab64-22e1e9ef50c1","Type":"ContainerDied","Data":"d0ed14b90cdea589c11fba79e9fc11afc5640cd6457d2bdd5661953101053d5e"}
Jun 02 15:47:12.425246 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:12.425213    2577 generic.go:358] "Generic (PLEG): container finished" podID="906be90d-c7ef-421a-ab64-22e1e9ef50c1" containerID="c66f66391b8f6a2650080d9876ae206809b4dd17937c17f2635260e53a1cc8c9" exitCode=0
Jun 02 15:47:12.425849 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:12.425255    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" event={"ID":"906be90d-c7ef-421a-ab64-22e1e9ef50c1","Type":"ContainerDied","Data":"c66f66391b8f6a2650080d9876ae206809b4dd17937c17f2635260e53a1cc8c9"}
Jun 02 15:47:13.262639 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:13.262595    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:13.262639 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:13.262645    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:13.262873 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:13.262750    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:13.262873 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:13.262762    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:13.262873 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:13.262808    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert podName:3384c3b6-30a7-4de5-86d6-a139400ecaf6 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:17.262789745 +0000 UTC m=+41.605210961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert") pod "ingress-canary-d6xhm" (UID: "3384c3b6-30a7-4de5-86d6-a139400ecaf6") : secret "canary-serving-cert" not found
Jun 02 15:47:13.262873 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:13.262823    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls podName:84dc34aa-0d48-4060-a21d-c93066f8c683 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:17.262817074 +0000 UTC m=+41.605238274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls") pod "dns-default-jllqh" (UID: "84dc34aa-0d48-4060-a21d-c93066f8c683") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:13.429859 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:13.429775    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" event={"ID":"906be90d-c7ef-421a-ab64-22e1e9ef50c1","Type":"ContainerStarted","Data":"002a8a14456593cd53fb31534966054b8c7a4ff22d44683ea5558657a091738e"}
Jun 02 15:47:13.454052 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:13.453997    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c6pv8" podStartSLOduration=3.791840923 podStartE2EDuration="37.453981994s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:46:37.426573983 +0000 UTC m=+1.768995183" lastFinishedPulling="2026-06-02 15:47:11.088715038 +0000 UTC m=+35.431136254" observedRunningTime="2026-06-02 15:47:13.45342405 +0000 UTC m=+37.795845268" watchObservedRunningTime="2026-06-02 15:47:13.453981994 +0000 UTC m=+37.796403220"
Jun 02 15:47:16.206840 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.206677    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74"]
Jun 02 15:47:16.242584 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.242548    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74"]
Jun 02 15:47:16.242734 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.242701    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74"
Jun 02 15:47:16.250193 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.250172    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jun 02 15:47:16.250307 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.250235    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q8kzc\""
Jun 02 15:47:16.250799 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.250782    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jun 02 15:47:16.385191 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.385149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fpl7\" (UniqueName: \"kubernetes.io/projected/8418f950-f099-4d7a-9991-1a13c03d241a-kube-api-access-7fpl7\") pod \"migrator-96c6f749-zgp74\" (UID: \"8418f950-f099-4d7a-9991-1a13c03d241a\") " pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74"
Jun 02 15:47:16.485913 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.485837    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7fpl7\" (UniqueName: \"kubernetes.io/projected/8418f950-f099-4d7a-9991-1a13c03d241a-kube-api-access-7fpl7\") pod \"migrator-96c6f749-zgp74\" (UID: \"8418f950-f099-4d7a-9991-1a13c03d241a\") " pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74"
Jun 02 15:47:16.499380 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.499351    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fpl7\" (UniqueName: \"kubernetes.io/projected/8418f950-f099-4d7a-9991-1a13c03d241a-kube-api-access-7fpl7\") pod \"migrator-96c6f749-zgp74\" (UID: \"8418f950-f099-4d7a-9991-1a13c03d241a\") " pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74"
Jun 02 15:47:16.552023 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.551988    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74"
Jun 02 15:47:16.709246 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:16.709213    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74"]
Jun 02 15:47:16.719729 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:16.719696    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8418f950_f099_4d7a_9991_1a13c03d241a.slice/crio-47e6c76980415bf1b193ab53789f19f2e8df8268e801d3d163e9aeda1233a02c WatchSource:0}: Error finding container 47e6c76980415bf1b193ab53789f19f2e8df8268e801d3d163e9aeda1233a02c: Status 404 returned error can't find the container with id 47e6c76980415bf1b193ab53789f19f2e8df8268e801d3d163e9aeda1233a02c
Jun 02 15:47:17.292697 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:17.292657    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:17.293150 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:17.292744    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:17.293150 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:17.292799    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:17.293150 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:17.292863    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:17.293150 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:17.292875    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert podName:3384c3b6-30a7-4de5-86d6-a139400ecaf6 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:25.292860113 +0000 UTC m=+49.635281323 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert") pod "ingress-canary-d6xhm" (UID: "3384c3b6-30a7-4de5-86d6-a139400ecaf6") : secret "canary-serving-cert" not found
Jun 02 15:47:17.293150 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:17.292914    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls podName:84dc34aa-0d48-4060-a21d-c93066f8c683 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:25.292898024 +0000 UTC m=+49.635319241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls") pod "dns-default-jllqh" (UID: "84dc34aa-0d48-4060-a21d-c93066f8c683") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:17.301775 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:17.301749    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2zxzn_46bdfe7b-9361-4645-bea3-a81b1c280953/dns-node-resolver/0.log"
Jun 02 15:47:17.439824 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:17.439778    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74" event={"ID":"8418f950-f099-4d7a-9991-1a13c03d241a","Type":"ContainerStarted","Data":"47e6c76980415bf1b193ab53789f19f2e8df8268e801d3d163e9aeda1233a02c"}
Jun 02 15:47:18.505714 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:18.505687    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mjvd5_497b35e9-18f2-40e6-939d-50cd84bf8ad4/node-ca/0.log"
Jun 02 15:47:19.446366 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:19.446330    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74" event={"ID":"8418f950-f099-4d7a-9991-1a13c03d241a","Type":"ContainerStarted","Data":"00faf5a1849ec909f9b662690d72dc9e45435b6c8c6b6d4b268da8f38565a7fd"}
Jun 02 15:47:19.446366 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:19.446368    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74" event={"ID":"8418f950-f099-4d7a-9991-1a13c03d241a","Type":"ContainerStarted","Data":"a34856420c6c854f0aa5078d21aa7efa18d105f69415f7a995632851a9bd9e6e"}
Jun 02 15:47:19.469034 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:19.468988    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-96c6f749-zgp74" podStartSLOduration=1.492586972 podStartE2EDuration="3.468974161s" podCreationTimestamp="2026-06-02 15:47:16 +0000 UTC" firstStartedPulling="2026-06-02 15:47:16.721492217 +0000 UTC m=+41.063913416" lastFinishedPulling="2026-06-02 15:47:18.697879393 +0000 UTC m=+43.040300605" observedRunningTime="2026-06-02 15:47:19.468852664 +0000 UTC m=+43.811273908" watchObservedRunningTime="2026-06-02 15:47:19.468974161 +0000 UTC m=+43.811395383"
Jun 02 15:47:25.357313 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:25.357275    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:25.357778 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:25.357360    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:25.357778 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:25.357455    2577 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
Jun 02 15:47:25.357778 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:25.357466    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 02 15:47:25.357778 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:25.357537    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls podName:84dc34aa-0d48-4060-a21d-c93066f8c683 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:41.357506608 +0000 UTC m=+65.699927824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls") pod "dns-default-jllqh" (UID: "84dc34aa-0d48-4060-a21d-c93066f8c683") : secret "dns-default-metrics-tls" not found
Jun 02 15:47:25.357778 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:25.357562    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert podName:3384c3b6-30a7-4de5-86d6-a139400ecaf6 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:41.357546269 +0000 UTC m=+65.699967469 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert") pod "ingress-canary-d6xhm" (UID: "3384c3b6-30a7-4de5-86d6-a139400ecaf6") : secret "canary-serving-cert" not found
Jun 02 15:47:34.414942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:34.414915    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xq8p6"
Jun 02 15:47:38.531347 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.531314    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-c65b8dc4b-l8vns"]
Jun 02 15:47:38.533978 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.533963    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-c65b8dc4b-l8vns"
Jun 02 15:47:38.535940 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.535914    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 02 15:47:38.536159 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.536145    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-l4qqb\""
Jun 02 15:47:38.536159 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.536151    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 02 15:47:38.547444 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.547418    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-c65b8dc4b-l8vns"]
Jun 02 15:47:38.550705 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.550682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wxxp\" (UniqueName: \"kubernetes.io/projected/3fc9bebc-7481-4323-9202-4f0a72ac4151-kube-api-access-6wxxp\") pod \"downloads-c65b8dc4b-l8vns\" (UID: \"3fc9bebc-7481-4323-9202-4f0a72ac4151\") " pod="openshift-console/downloads-c65b8dc4b-l8vns"
Jun 02 15:47:38.651769 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.651731    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6wxxp\" (UniqueName: \"kubernetes.io/projected/3fc9bebc-7481-4323-9202-4f0a72ac4151-kube-api-access-6wxxp\") pod \"downloads-c65b8dc4b-l8vns\" (UID: \"3fc9bebc-7481-4323-9202-4f0a72ac4151\") " pod="openshift-console/downloads-c65b8dc4b-l8vns"
Jun 02 15:47:38.656635 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.656609    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tl27z"]
Jun 02 15:47:38.658461 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.658442    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.661015 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.660993    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 02 15:47:38.661161 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.660993    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 02 15:47:38.661161 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.661039    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 02 15:47:38.661279 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.661265    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 02 15:47:38.661364 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.661340    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tl6r5\""
Jun 02 15:47:38.675370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.675347    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tl27z"]
Jun 02 15:47:38.678844 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.678812    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wxxp\" (UniqueName: \"kubernetes.io/projected/3fc9bebc-7481-4323-9202-4f0a72ac4151-kube-api-access-6wxxp\") pod \"downloads-c65b8dc4b-l8vns\" (UID: \"3fc9bebc-7481-4323-9202-4f0a72ac4151\") " pod="openshift-console/downloads-c65b8dc4b-l8vns"
Jun 02 15:47:38.752296 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.752262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84cb8848-73cd-40a0-902a-11fc078cff29-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.752512 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.752316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84cb8848-73cd-40a0-902a-11fc078cff29-crio-socket\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.752512 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.752333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84cb8848-73cd-40a0-902a-11fc078cff29-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.752512 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.752430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84cb8848-73cd-40a0-902a-11fc078cff29-data-volume\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.752512 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.752466    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8jjt\" (UniqueName: \"kubernetes.io/projected/84cb8848-73cd-40a0-902a-11fc078cff29-kube-api-access-p8jjt\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.842222 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.842135    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-c65b8dc4b-l8vns"
Jun 02 15:47:38.853768 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.853741    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84cb8848-73cd-40a0-902a-11fc078cff29-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.853880 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.853799    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84cb8848-73cd-40a0-902a-11fc078cff29-crio-socket\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.853880 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.853823    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84cb8848-73cd-40a0-902a-11fc078cff29-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.853880 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.853858    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84cb8848-73cd-40a0-902a-11fc078cff29-data-volume\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.854047 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.853887    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p8jjt\" (UniqueName: \"kubernetes.io/projected/84cb8848-73cd-40a0-902a-11fc078cff29-kube-api-access-p8jjt\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.854099 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.854079    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/84cb8848-73cd-40a0-902a-11fc078cff29-crio-socket\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.854549 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.854529    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/84cb8848-73cd-40a0-902a-11fc078cff29-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.854972 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.854953    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/84cb8848-73cd-40a0-902a-11fc078cff29-data-volume\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.856434 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.856414    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/84cb8848-73cd-40a0-902a-11fc078cff29-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.863973 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.863951    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8jjt\" (UniqueName: \"kubernetes.io/projected/84cb8848-73cd-40a0-902a-11fc078cff29-kube-api-access-p8jjt\") pod \"insights-runtime-extractor-tl27z\" (UID: \"84cb8848-73cd-40a0-902a-11fc078cff29\") " pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.966642 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.966596    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tl27z"
Jun 02 15:47:38.972662 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:38.972630    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-c65b8dc4b-l8vns"]
Jun 02 15:47:38.976989 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:38.976963    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc9bebc_7481_4323_9202_4f0a72ac4151.slice/crio-9e78c410e303eaf0cbe2836256b6f7a832e180a959f7de3fe17bbebea09cf8fc WatchSource:0}: Error finding container 9e78c410e303eaf0cbe2836256b6f7a832e180a959f7de3fe17bbebea09cf8fc: Status 404 returned error can't find the container with id 9e78c410e303eaf0cbe2836256b6f7a832e180a959f7de3fe17bbebea09cf8fc
Jun 02 15:47:39.097422 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:39.097326    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tl27z"]
Jun 02 15:47:39.100043 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:39.100009    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cb8848_73cd_40a0_902a_11fc078cff29.slice/crio-05d838cf303a885d53b12fab57d31b2071ecaeae774b38b470e3d4120b36b9e5 WatchSource:0}: Error finding container 05d838cf303a885d53b12fab57d31b2071ecaeae774b38b470e3d4120b36b9e5: Status 404 returned error can't find the container with id 05d838cf303a885d53b12fab57d31b2071ecaeae774b38b470e3d4120b36b9e5
Jun 02 15:47:39.485662 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:39.485626    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tl27z" event={"ID":"84cb8848-73cd-40a0-902a-11fc078cff29","Type":"ContainerStarted","Data":"2d4cb62d0fdaa4cd0a311a0ddc35891ef1c9fd6559c1fe0f8ee3d6e7fafd6f77"}
Jun 02 15:47:39.485836 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:39.485671    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tl27z" event={"ID":"84cb8848-73cd-40a0-902a-11fc078cff29","Type":"ContainerStarted","Data":"05d838cf303a885d53b12fab57d31b2071ecaeae774b38b470e3d4120b36b9e5"}
Jun 02 15:47:39.486552 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:39.486529    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-c65b8dc4b-l8vns" event={"ID":"3fc9bebc-7481-4323-9202-4f0a72ac4151","Type":"ContainerStarted","Data":"9e78c410e303eaf0cbe2836256b6f7a832e180a959f7de3fe17bbebea09cf8fc"}
Jun 02 15:47:39.983120 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:39.983083    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"]
Jun 02 15:47:39.986565 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:39.986546    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"
Jun 02 15:47:39.989264 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:39.989238    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-csmcr\""
Jun 02 15:47:39.989264 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:39.989261    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jun 02 15:47:40.001825 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.001801    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"]
Jun 02 15:47:40.063979 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.063909    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ab578e0a-45c7-4be1-ac08-8fa0559acb54-tls-certificates\") pod \"prometheus-operator-admission-webhook-59b7f8756b-m2h8s\" (UID: \"ab578e0a-45c7-4be1-ac08-8fa0559acb54\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"
Jun 02 15:47:40.165203 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.165172    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ab578e0a-45c7-4be1-ac08-8fa0559acb54-tls-certificates\") pod \"prometheus-operator-admission-webhook-59b7f8756b-m2h8s\" (UID: \"ab578e0a-45c7-4be1-ac08-8fa0559acb54\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"
Jun 02 15:47:40.168183 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.168152    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ab578e0a-45c7-4be1-ac08-8fa0559acb54-tls-certificates\") pod \"prometheus-operator-admission-webhook-59b7f8756b-m2h8s\" (UID: \"ab578e0a-45c7-4be1-ac08-8fa0559acb54\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"
Jun 02 15:47:40.296586 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.296550    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"
Jun 02 15:47:40.441588 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.441552    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"]
Jun 02 15:47:40.446372 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:40.446341    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab578e0a_45c7_4be1_ac08_8fa0559acb54.slice/crio-6dedb0eca6f33a259e03385c81e0206392b06db91a8f0bdf6566f77c5f019a69 WatchSource:0}: Error finding container 6dedb0eca6f33a259e03385c81e0206392b06db91a8f0bdf6566f77c5f019a69: Status 404 returned error can't find the container with id 6dedb0eca6f33a259e03385c81e0206392b06db91a8f0bdf6566f77c5f019a69
Jun 02 15:47:40.490826 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.490784    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tl27z" event={"ID":"84cb8848-73cd-40a0-902a-11fc078cff29","Type":"ContainerStarted","Data":"ed0e7bd81bfdee180d8718b3d96408d9f560b5e209a3e3c52854f8821389a187"}
Jun 02 15:47:40.492058 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.492007    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s" event={"ID":"ab578e0a-45c7-4be1-ac08-8fa0559acb54","Type":"ContainerStarted","Data":"6dedb0eca6f33a259e03385c81e0206392b06db91a8f0bdf6566f77c5f019a69"}
Jun 02 15:47:40.871181 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.871144    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:40.873743 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.873720    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 02 15:47:40.883716 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.883693    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f13e68f4-6625-437f-bb2c-974057826f5c-metrics-certs\") pod \"network-metrics-daemon-mdgjl\" (UID: \"f13e68f4-6625-437f-bb2c-974057826f5c\") " pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:40.936115 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.936065    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2vmzq\""
Jun 02 15:47:40.944806 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.944777    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mdgjl"
Jun 02 15:47:40.972131 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.972091    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:40.975070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.975043    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 02 15:47:40.985745 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.985721    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 02 15:47:40.996832 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:40.996778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbfgm\" (UniqueName: \"kubernetes.io/projected/91fef626-9a99-4972-8f0f-59ee18f994ec-kube-api-access-sbfgm\") pod \"network-check-target-wp2zq\" (UID: \"91fef626-9a99-4972-8f0f-59ee18f994ec\") " pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:41.104380 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.104318    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mdgjl"]
Jun 02 15:47:41.108905 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:41.108880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13e68f4_6625_437f_bb2c_974057826f5c.slice/crio-cc82e1f988ff39efbb19b3bb5742ecfcb842f15ff716541be8af11e12ed97561 WatchSource:0}: Error finding container cc82e1f988ff39efbb19b3bb5742ecfcb842f15ff716541be8af11e12ed97561: Status 404 returned error can't find the container with id cc82e1f988ff39efbb19b3bb5742ecfcb842f15ff716541be8af11e12ed97561
Jun 02 15:47:41.231310 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.231280    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6qg5h\""
Jun 02 15:47:41.239600 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.239422    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:41.374518 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.374477    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:41.374700 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.374547    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:41.377322 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.377296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84dc34aa-0d48-4060-a21d-c93066f8c683-metrics-tls\") pod \"dns-default-jllqh\" (UID: \"84dc34aa-0d48-4060-a21d-c93066f8c683\") " pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:41.378412 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.378349    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3384c3b6-30a7-4de5-86d6-a139400ecaf6-cert\") pod \"ingress-canary-d6xhm\" (UID: \"3384c3b6-30a7-4de5-86d6-a139400ecaf6\") " pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:41.392057 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.392010    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wp2zq"]
Jun 02 15:47:41.496146 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.496051    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdgjl" event={"ID":"f13e68f4-6625-437f-bb2c-974057826f5c","Type":"ContainerStarted","Data":"cc82e1f988ff39efbb19b3bb5742ecfcb842f15ff716541be8af11e12ed97561"}
Jun 02 15:47:41.647834 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.647810    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-g6cp8\""
Jun 02 15:47:41.656564 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.656539    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:41.662362 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:41.662337    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fef626_9a99_4972_8f0f_59ee18f994ec.slice/crio-e8e374cf6e7e4c3f82d6a398a07c2967eb236f948cf356bf5bfe8c52b1dd2711 WatchSource:0}: Error finding container e8e374cf6e7e4c3f82d6a398a07c2967eb236f948cf356bf5bfe8c52b1dd2711: Status 404 returned error can't find the container with id e8e374cf6e7e4c3f82d6a398a07c2967eb236f948cf356bf5bfe8c52b1dd2711
Jun 02 15:47:41.664451 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.664421    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ws8nx\""
Jun 02 15:47:41.672125 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:41.672107    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-d6xhm"
Jun 02 15:47:42.499290 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:42.499253    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wp2zq" event={"ID":"91fef626-9a99-4972-8f0f-59ee18f994ec","Type":"ContainerStarted","Data":"e8e374cf6e7e4c3f82d6a398a07c2967eb236f948cf356bf5bfe8c52b1dd2711"}
Jun 02 15:47:42.846937 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:42.846363    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jllqh"]
Jun 02 15:47:42.851468 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:42.851399    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84dc34aa_0d48_4060_a21d_c93066f8c683.slice/crio-fdc92fe07ba1ac47e2bbb82f2859d259d60c5f5600283a49c3a9b77a118494b1 WatchSource:0}: Error finding container fdc92fe07ba1ac47e2bbb82f2859d259d60c5f5600283a49c3a9b77a118494b1: Status 404 returned error can't find the container with id fdc92fe07ba1ac47e2bbb82f2859d259d60c5f5600283a49c3a9b77a118494b1
Jun 02 15:47:42.864969 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:42.864919    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-d6xhm"]
Jun 02 15:47:42.871016 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:42.870991    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3384c3b6_30a7_4de5_86d6_a139400ecaf6.slice/crio-9e38eae40e181f7eab4af74ceb86a7cb52f82b33aeba17409be5a201dc89001d WatchSource:0}: Error finding container 9e38eae40e181f7eab4af74ceb86a7cb52f82b33aeba17409be5a201dc89001d: Status 404 returned error can't find the container with id 9e38eae40e181f7eab4af74ceb86a7cb52f82b33aeba17409be5a201dc89001d
Jun 02 15:47:43.505266 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:43.505222    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d6xhm" event={"ID":"3384c3b6-30a7-4de5-86d6-a139400ecaf6","Type":"ContainerStarted","Data":"9e38eae40e181f7eab4af74ceb86a7cb52f82b33aeba17409be5a201dc89001d"}
Jun 02 15:47:43.508446 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:43.507609    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tl27z" event={"ID":"84cb8848-73cd-40a0-902a-11fc078cff29","Type":"ContainerStarted","Data":"100c37b1520d959ebe9eb3cf0703b21fad0ae619bb2f7b8e4970edd9ece7e4b7"}
Jun 02 15:47:43.509677 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:43.509651    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jllqh" event={"ID":"84dc34aa-0d48-4060-a21d-c93066f8c683","Type":"ContainerStarted","Data":"fdc92fe07ba1ac47e2bbb82f2859d259d60c5f5600283a49c3a9b77a118494b1"}
Jun 02 15:47:43.512124 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:43.511782    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s" event={"ID":"ab578e0a-45c7-4be1-ac08-8fa0559acb54","Type":"ContainerStarted","Data":"adca5d474f3b6b06bcd618814e29cbde22fc9e245b5a246c661625a19a2290d0"}
Jun 02 15:47:43.512462 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:43.512427    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"
Jun 02 15:47:43.517823 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:43.517780    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s"
Jun 02 15:47:43.555489 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:43.555423    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tl27z" podStartSLOduration=2.019714205 podStartE2EDuration="5.555379086s" podCreationTimestamp="2026-06-02 15:47:38 +0000 UTC" firstStartedPulling="2026-06-02 15:47:39.146252908 +0000 UTC m=+63.488674108" lastFinishedPulling="2026-06-02 15:47:42.681917774 +0000 UTC m=+67.024338989" observedRunningTime="2026-06-02 15:47:43.535181414 +0000 UTC m=+67.877602636" watchObservedRunningTime="2026-06-02 15:47:43.555379086 +0000 UTC m=+67.897800308"
Jun 02 15:47:44.519333 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:44.519039    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdgjl" event={"ID":"f13e68f4-6625-437f-bb2c-974057826f5c","Type":"ContainerStarted","Data":"1ac18b14a15f8668a659700514b2690bada0deba0ad05f97aecd39dee39e6565"}
Jun 02 15:47:44.519333 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:44.519082    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mdgjl" event={"ID":"f13e68f4-6625-437f-bb2c-974057826f5c","Type":"ContainerStarted","Data":"165fb6ebae2a6438e19bd726c75594547dc208c93ed66a021950c1a632845e56"}
Jun 02 15:47:44.544430 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:44.543967    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-59b7f8756b-m2h8s" podStartSLOduration=3.317197757 podStartE2EDuration="5.543947509s" podCreationTimestamp="2026-06-02 15:47:39 +0000 UTC" firstStartedPulling="2026-06-02 15:47:40.448752654 +0000 UTC m=+64.791173857" lastFinishedPulling="2026-06-02 15:47:42.675502403 +0000 UTC m=+67.017923609" observedRunningTime="2026-06-02 15:47:43.555791103 +0000 UTC m=+67.898212315" watchObservedRunningTime="2026-06-02 15:47:44.543947509 +0000 UTC m=+68.886368732"
Jun 02 15:47:46.246407 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.245950    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mdgjl" podStartSLOduration=67.697036251 podStartE2EDuration="1m10.245930685s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:47:41.111200468 +0000 UTC m=+65.453621668" lastFinishedPulling="2026-06-02 15:47:43.660094893 +0000 UTC m=+68.002516102" observedRunningTime="2026-06-02 15:47:44.544414569 +0000 UTC m=+68.886835792" watchObservedRunningTime="2026-06-02 15:47:46.245930685 +0000 UTC m=+70.588351908"
Jun 02 15:47:46.526518 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.526432    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wp2zq" event={"ID":"91fef626-9a99-4972-8f0f-59ee18f994ec","Type":"ContainerStarted","Data":"f8f299fd68ac5f52b3b3e47f1ab59915ae92ec6a776e380b2523612b12efc24d"}
Jun 02 15:47:46.526700 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.526677    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:47:46.527999 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.527970    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-d6xhm" event={"ID":"3384c3b6-30a7-4de5-86d6-a139400ecaf6","Type":"ContainerStarted","Data":"00e059f66b301ac2e282a4ddfc8b4cc04641dfab392987d1bb74970b426ae04c"}
Jun 02 15:47:46.529702 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.529675    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jllqh" event={"ID":"84dc34aa-0d48-4060-a21d-c93066f8c683","Type":"ContainerStarted","Data":"dc3abb5f7b31b6533c280bed74a923d30cbd0f6fb1ca9decd985018f21bbf212"}
Jun 02 15:47:46.529822 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.529707    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jllqh" event={"ID":"84dc34aa-0d48-4060-a21d-c93066f8c683","Type":"ContainerStarted","Data":"83a99034c57d00eba0da604ba11e27faf2be4c7374d94ec158ca9fe6aad80558"}
Jun 02 15:47:46.529822 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.529808    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jllqh"
Jun 02 15:47:46.547873 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.547826    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wp2zq" podStartSLOduration=65.980414812 podStartE2EDuration="1m10.547810127s" podCreationTimestamp="2026-06-02 15:46:36 +0000 UTC" firstStartedPulling="2026-06-02 15:47:41.664121798 +0000 UTC m=+66.006542999" lastFinishedPulling="2026-06-02 15:47:46.2315171 +0000 UTC m=+70.573938314" observedRunningTime="2026-06-02 15:47:46.546267851 +0000 UTC m=+70.888689072" watchObservedRunningTime="2026-06-02 15:47:46.547810127 +0000 UTC m=+70.890231349"
Jun 02 15:47:46.572508 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.572452    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-jllqh" podStartSLOduration=34.212289296 podStartE2EDuration="37.572437947s" podCreationTimestamp="2026-06-02 15:47:09 +0000 UTC" firstStartedPulling="2026-06-02 15:47:42.855502249 +0000 UTC m=+67.197923454" lastFinishedPulling="2026-06-02 15:47:46.215650905 +0000 UTC m=+70.558072105" observedRunningTime="2026-06-02 15:47:46.572148776 +0000 UTC m=+70.914569998" watchObservedRunningTime="2026-06-02 15:47:46.572437947 +0000 UTC m=+70.914859169"
Jun 02 15:47:46.596360 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:46.596305    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-d6xhm" podStartSLOduration=34.249542809 podStartE2EDuration="37.596286032s" podCreationTimestamp="2026-06-02 15:47:09 +0000 UTC" firstStartedPulling="2026-06-02 15:47:42.872984488 +0000 UTC m=+67.215405692" lastFinishedPulling="2026-06-02 15:47:46.219727695 +0000 UTC m=+70.562148915" observedRunningTime="2026-06-02 15:47:46.595289714 +0000 UTC m=+70.937710937" watchObservedRunningTime="2026-06-02 15:47:46.596286032 +0000 UTC m=+70.938707257"
Jun 02 15:47:49.726951 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.726909    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"]
Jun 02 15:47:49.731489 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.731455    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.734132 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.734107    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jun 02 15:47:49.734277 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.734115    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4fqrt"]
Jun 02 15:47:49.734340 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.734317    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jun 02 15:47:49.734989 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.734625    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-j9kfr\""
Jun 02 15:47:49.734989 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.734627    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 02 15:47:49.734989 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.734846    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 02 15:47:49.734989 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.734989    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 02 15:47:49.738336 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.738315    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.740594 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.740574    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 02 15:47:49.740684 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.740574    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 02 15:47:49.740751 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.740733    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 02 15:47:49.740882 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.740865    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zfv5r\""
Jun 02 15:47:49.753736 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.753713    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"]
Jun 02 15:47:49.775592 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.775557    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"]
Jun 02 15:47:49.780071 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.780047    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.783207 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.783176    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jun 02 15:47:49.783338 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.783306    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jun 02 15:47:49.783499 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.783481    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-bmt6f\""
Jun 02 15:47:49.783604 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.783525    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jun 02 15:47:49.792674 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.792649    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"]
Jun 02 15:47:49.834710 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-root\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.834865 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.834865 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-tls\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.834865 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.834865 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dfe68d2-3070-49c4-a420-e5b9dd76322c-metrics-client-ca\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.834865 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.835024 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834889    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.835024 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.835024 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-sys\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.835024 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmcj\" (UniqueName: \"kubernetes.io/projected/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-api-access-2mmcj\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.835024 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.834985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqfgq\" (UniqueName: \"kubernetes.io/projected/cf51054f-3332-4b3c-8396-67be19d8bb24-kube-api-access-qqfgq\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.835024 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.835018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-textfile\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.835214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.835042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf51054f-3332-4b3c-8396-67be19d8bb24-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.835214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.835060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa081bae-80b7-4e42-bb44-69311dbb44cc-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.835214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.835106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fa081bae-80b7-4e42-bb44-69311dbb44cc-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.835214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.835132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwfkm\" (UniqueName: \"kubernetes.io/projected/2dfe68d2-3070-49c4-a420-e5b9dd76322c-kube-api-access-lwfkm\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.835214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.835163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.835214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.835185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.835401 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.835242    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-wtmp\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936420    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmcj\" (UniqueName: \"kubernetes.io/projected/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-api-access-2mmcj\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.936460 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936462    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qqfgq\" (UniqueName: \"kubernetes.io/projected/cf51054f-3332-4b3c-8396-67be19d8bb24-kube-api-access-qqfgq\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.936711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936483    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-textfile\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936502    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf51054f-3332-4b3c-8396-67be19d8bb24-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.936711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936520    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa081bae-80b7-4e42-bb44-69311dbb44cc-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.936711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936543    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fa081bae-80b7-4e42-bb44-69311dbb44cc-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.936711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936560    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lwfkm\" (UniqueName: \"kubernetes.io/projected/2dfe68d2-3070-49c4-a420-e5b9dd76322c-kube-api-access-lwfkm\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936583    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936772    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936814    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-wtmp\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936848    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-root\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936882    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936907    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-tls\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936932    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936935    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-textfile\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936965    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dfe68d2-3070-49c4-a420-e5b9dd76322c-metrics-client-ca\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.936997 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936987    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-wtmp\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.936991    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937011    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fa081bae-80b7-4e42-bb44-69311dbb44cc-volume-directive-shadow\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937024    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-root\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937051    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937081    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937110    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-sys\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:49.937119    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2dfe68d2-3070-49c4-a420-e5b9dd76322c-sys\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:49.937187    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-tls podName:2dfe68d2-3070-49c4-a420-e5b9dd76322c nodeName:}" failed. No retries permitted until 2026-06-02 15:47:50.437167572 +0000 UTC m=+74.779588773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-tls") pod "node-exporter-4fqrt" (UID: "2dfe68d2-3070-49c4-a420-e5b9dd76322c") : secret "node-exporter-tls" not found
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:49.937244    2577 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:49.937301    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-tls podName:fa081bae-80b7-4e42-bb44-69311dbb44cc nodeName:}" failed. No retries permitted until 2026-06-02 15:47:50.437284215 +0000 UTC m=+74.779705415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-tls") pod "kube-state-metrics-8469c486bb-sgxtg" (UID: "fa081bae-80b7-4e42-bb44-69311dbb44cc") : secret "kube-state-metrics-tls" not found
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:49.937357    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937373    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf51054f-3332-4b3c-8396-67be19d8bb24-metrics-client-ca\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.937474 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:49.937415    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-tls podName:cf51054f-3332-4b3c-8396-67be19d8bb24 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:50.437377388 +0000 UTC m=+74.779798591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-tls") pod "openshift-state-metrics-85658796db-rvlz9" (UID: "cf51054f-3332-4b3c-8396-67be19d8bb24") : secret "openshift-state-metrics-tls" not found
Jun 02 15:47:49.938083 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937494    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.938083 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937518    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa081bae-80b7-4e42-bb44-69311dbb44cc-metrics-client-ca\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.938083 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.937878    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2dfe68d2-3070-49c4-a420-e5b9dd76322c-metrics-client-ca\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.938083 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.938017    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.939505 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.939482    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:49.939599 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.939543    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.939638 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.939596    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.957042 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.957016    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mmcj\" (UniqueName: \"kubernetes.io/projected/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-api-access-2mmcj\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:49.957169 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.957147    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqfgq\" (UniqueName: \"kubernetes.io/projected/cf51054f-3332-4b3c-8396-67be19d8bb24-kube-api-access-qqfgq\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:49.957267 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:49.957249    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwfkm\" (UniqueName: \"kubernetes.io/projected/2dfe68d2-3070-49c4-a420-e5b9dd76322c-kube-api-access-lwfkm\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:50.440940 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.440903    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-tls\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:50.441151 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.440972    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:50.441151 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.441005    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:50.443782 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.443736    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2dfe68d2-3070-49c4-a420-e5b9dd76322c-node-exporter-tls\") pod \"node-exporter-4fqrt\" (UID: \"2dfe68d2-3070-49c4-a420-e5b9dd76322c\") " pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:50.443931 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.443800    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa081bae-80b7-4e42-bb44-69311dbb44cc-kube-state-metrics-tls\") pod \"kube-state-metrics-8469c486bb-sgxtg\" (UID: \"fa081bae-80b7-4e42-bb44-69311dbb44cc\") " pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:50.443931 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.443884    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf51054f-3332-4b3c-8396-67be19d8bb24-openshift-state-metrics-tls\") pod \"openshift-state-metrics-85658796db-rvlz9\" (UID: \"cf51054f-3332-4b3c-8396-67be19d8bb24\") " pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:50.643982 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.643946    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"
Jun 02 15:47:50.649798 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.649766    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4fqrt"
Jun 02 15:47:50.657254 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:50.657228    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfe68d2_3070_49c4_a420_e5b9dd76322c.slice/crio-b2831d60aee28ad9d5249460f18facd72f2fa613987cfc87f3d9377d554456c1 WatchSource:0}: Error finding container b2831d60aee28ad9d5249460f18facd72f2fa613987cfc87f3d9377d554456c1: Status 404 returned error can't find the container with id b2831d60aee28ad9d5249460f18facd72f2fa613987cfc87f3d9377d554456c1
Jun 02 15:47:50.682025 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.681994    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:47:50.688049 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.688028    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.690889 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.690500    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jun 02 15:47:50.690889 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.690826    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jun 02 15:47:50.691269 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.691056    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jun 02 15:47:50.692035 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.692015    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jun 02 15:47:50.692208 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.692153    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jun 02 15:47:50.692285 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.692015    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cd5f5\""
Jun 02 15:47:50.692285 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.692237    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jun 02 15:47:50.692458 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.692442    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"
Jun 02 15:47:50.692538 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.692480    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jun 02 15:47:50.693861 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.692905    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jun 02 15:47:50.693861 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.693237    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jun 02 15:47:50.713450 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.713370    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:47:50.749988 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.749952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-out\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750377 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750377 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750171    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750377 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750202    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750377 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750290    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98fl\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-kube-api-access-k98fl\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750464    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-volume\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750589    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-web-config\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.750977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.750685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.823516 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.823479    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-85658796db-rvlz9"]
Jun 02 15:47:50.826699 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:50.826672    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf51054f_3332_4b3c_8396_67be19d8bb24.slice/crio-a6a9274d3667ea2868b304db784a643e645a7d4397b0e3d50e386dd9ec01f36c WatchSource:0}: Error finding container a6a9274d3667ea2868b304db784a643e645a7d4397b0e3d50e386dd9ec01f36c: Status 404 returned error can't find the container with id a6a9274d3667ea2868b304db784a643e645a7d4397b0e3d50e386dd9ec01f36c
Jun 02 15:47:50.851091 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851058    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851125    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851164    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-volume\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851214 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851193    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851375 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851220    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-web-config\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851375 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851249    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851375 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:50.851322    2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jun 02 15:47:50.851528 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:50.851424    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls podName:0252993b-dcf8-44cb-92ca-4c25ab4eeb06 nodeName:}" failed. No retries permitted until 2026-06-02 15:47:51.351400266 +0000 UTC m=+75.693821481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06") : secret "alertmanager-main-tls" not found
Jun 02 15:47:50.851588 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851534    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851588 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851578    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-out\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851687 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851611    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851687 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851656    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851787 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851685    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851787 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851719    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-k98fl\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-kube-api-access-k98fl\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.851787 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.851757    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.852765 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.852170    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.852765 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.852558    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.853698 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.853653    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.854352 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.854322    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-web-config\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.855351 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.854656    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.856247 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.855605    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-out\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.856247 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.855804    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-volume\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.856247 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.856190    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.856247 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.856226    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.856542 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.856416    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.857494 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.857476    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.865068 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.865023    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg"]
Jun 02 15:47:50.871197 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:50.871173    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98fl\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-kube-api-access-k98fl\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:50.873605 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:50.873579    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa081bae_80b7_4e42_bb44_69311dbb44cc.slice/crio-a3a78009b5161e5c44b0ba5d547f8c54cdeb2bab57e0278d8ad69a596140901b WatchSource:0}: Error finding container a3a78009b5161e5c44b0ba5d547f8c54cdeb2bab57e0278d8ad69a596140901b: Status 404 returned error can't find the container with id a3a78009b5161e5c44b0ba5d547f8c54cdeb2bab57e0278d8ad69a596140901b
Jun 02 15:47:51.356333 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.356236    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:51.359171 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.359132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:51.546171 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.546140    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg" event={"ID":"fa081bae-80b7-4e42-bb44-69311dbb44cc","Type":"ContainerStarted","Data":"a3a78009b5161e5c44b0ba5d547f8c54cdeb2bab57e0278d8ad69a596140901b"}
Jun 02 15:47:51.549633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.549600    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fqrt" event={"ID":"2dfe68d2-3070-49c4-a420-e5b9dd76322c","Type":"ContainerStarted","Data":"b2831d60aee28ad9d5249460f18facd72f2fa613987cfc87f3d9377d554456c1"}
Jun 02 15:47:51.550977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.550951    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9" event={"ID":"cf51054f-3332-4b3c-8396-67be19d8bb24","Type":"ContainerStarted","Data":"87d3c1aec33f4fb7e33fdd83d1120a29c82f8cb04faa7f5ba4ffca3f1ce84a67"}
Jun 02 15:47:51.551081 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.550982    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9" event={"ID":"cf51054f-3332-4b3c-8396-67be19d8bb24","Type":"ContainerStarted","Data":"f59dc3a5d52e179d4296e29264003074a31693c0f28dc0d7719ca5f786605a8a"}
Jun 02 15:47:51.551081 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.550996    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9" event={"ID":"cf51054f-3332-4b3c-8396-67be19d8bb24","Type":"ContainerStarted","Data":"a6a9274d3667ea2868b304db784a643e645a7d4397b0e3d50e386dd9ec01f36c"}
Jun 02 15:47:51.601572 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.601548    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:47:51.788495 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:51.788462    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:47:51.804452 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:47:51.804418    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0252993b_dcf8_44cb_92ca_4c25ab4eeb06.slice/crio-bf6776558bceffe0de239d06270efe5eb14fc9a1dc6f1ef7af63fc6a64916428 WatchSource:0}: Error finding container bf6776558bceffe0de239d06270efe5eb14fc9a1dc6f1ef7af63fc6a64916428: Status 404 returned error can't find the container with id bf6776558bceffe0de239d06270efe5eb14fc9a1dc6f1ef7af63fc6a64916428
Jun 02 15:47:52.558315 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:52.557565    2577 generic.go:358] "Generic (PLEG): container finished" podID="2dfe68d2-3070-49c4-a420-e5b9dd76322c" containerID="5590a5d00828a8d65e7fed6d85b41f5a320792635521eeb2bc4d968f172eda6a" exitCode=0
Jun 02 15:47:52.558315 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:52.557811    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fqrt" event={"ID":"2dfe68d2-3070-49c4-a420-e5b9dd76322c","Type":"ContainerDied","Data":"5590a5d00828a8d65e7fed6d85b41f5a320792635521eeb2bc4d968f172eda6a"}
Jun 02 15:47:52.559423 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:52.559380    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerStarted","Data":"bf6776558bceffe0de239d06270efe5eb14fc9a1dc6f1ef7af63fc6a64916428"}
Jun 02 15:47:53.563493 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.563401    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9" event={"ID":"cf51054f-3332-4b3c-8396-67be19d8bb24","Type":"ContainerStarted","Data":"8679783238f87f8265fb2b01293731f281ef10b436414ad686f9f496dc97f991"}
Jun 02 15:47:53.565443 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.565413    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg" event={"ID":"fa081bae-80b7-4e42-bb44-69311dbb44cc","Type":"ContainerStarted","Data":"72c1a3e67f06ea165b28e829fe59df82b477510e0a678a8ca3cd4f2e457620a1"}
Jun 02 15:47:53.565575 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.565447    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg" event={"ID":"fa081bae-80b7-4e42-bb44-69311dbb44cc","Type":"ContainerStarted","Data":"9fad6b3d84c1bebff785af05de5601b51abe944d2d43fb8c5ebedc57e5a739fb"}
Jun 02 15:47:53.565575 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.565461    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg" event={"ID":"fa081bae-80b7-4e42-bb44-69311dbb44cc","Type":"ContainerStarted","Data":"4623c2b8da5e37b5e2e8425c9b7e4a5ea729e15732015c32bba298478661f428"}
Jun 02 15:47:53.567181 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.567162    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fqrt" event={"ID":"2dfe68d2-3070-49c4-a420-e5b9dd76322c","Type":"ContainerStarted","Data":"2cd460ccb985c994bb5fb0a7c98ee7acf0de73b83c9663842f8d38fdce71a285"}
Jun 02 15:47:53.567286 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.567187    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fqrt" event={"ID":"2dfe68d2-3070-49c4-a420-e5b9dd76322c","Type":"ContainerStarted","Data":"c9e60b996a30f5dd35b41d271a035b60cf4464127fc884220cd7c6e2f9350023"}
Jun 02 15:47:53.568507 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.568486    2577 generic.go:358] "Generic (PLEG): container finished" podID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerID="8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078" exitCode=0
Jun 02 15:47:53.568579 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.568534    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerDied","Data":"8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078"}
Jun 02 15:47:53.586065 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.586017    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-85658796db-rvlz9" podStartSLOduration=3.058006899 podStartE2EDuration="4.586005221s" podCreationTimestamp="2026-06-02 15:47:49 +0000 UTC" firstStartedPulling="2026-06-02 15:47:50.959803202 +0000 UTC m=+75.302224405" lastFinishedPulling="2026-06-02 15:47:52.487801515 +0000 UTC m=+76.830222727" observedRunningTime="2026-06-02 15:47:53.585114454 +0000 UTC m=+77.927535678" watchObservedRunningTime="2026-06-02 15:47:53.586005221 +0000 UTC m=+77.928426442"
Jun 02 15:47:53.636976 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.636922    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4fqrt" podStartSLOduration=3.743050992 podStartE2EDuration="4.636900995s" podCreationTimestamp="2026-06-02 15:47:49 +0000 UTC" firstStartedPulling="2026-06-02 15:47:50.659094348 +0000 UTC m=+75.001515563" lastFinishedPulling="2026-06-02 15:47:51.552944365 +0000 UTC m=+75.895365566" observedRunningTime="2026-06-02 15:47:53.635612731 +0000 UTC m=+77.978033954" watchObservedRunningTime="2026-06-02 15:47:53.636900995 +0000 UTC m=+77.979322222"
Jun 02 15:47:53.657819 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:53.657770    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-8469c486bb-sgxtg" podStartSLOduration=3.047674326 podStartE2EDuration="4.657755557s" podCreationTimestamp="2026-06-02 15:47:49 +0000 UTC" firstStartedPulling="2026-06-02 15:47:50.876075055 +0000 UTC m=+75.218496259" lastFinishedPulling="2026-06-02 15:47:52.486156282 +0000 UTC m=+76.828577490" observedRunningTime="2026-06-02 15:47:53.656096118 +0000 UTC m=+77.998517341" watchObservedRunningTime="2026-06-02 15:47:53.657755557 +0000 UTC m=+78.000176779"
Jun 02 15:47:54.227828 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.227795    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67ddbc6856-ws4wh"]
Jun 02 15:47:54.232466 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.232443    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.234972 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.234946    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f4cbn83238oc8\""
Jun 02 15:47:54.235712 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.235562    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jun 02 15:47:54.235712 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.235599    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-ksdtn\""
Jun 02 15:47:54.235712 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.235596    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jun 02 15:47:54.235934 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.235715    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jun 02 15:47:54.235934 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.235854    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jun 02 15:47:54.244586 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.244564    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67ddbc6856-ws4wh"]
Jun 02 15:47:54.278502 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.278467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e6a477b5-895e-4688-ad78-cb2c0ee2422e-audit-log\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.278673 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.278526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xsh\" (UniqueName: \"kubernetes.io/projected/e6a477b5-895e-4688-ad78-cb2c0ee2422e-kube-api-access-b8xsh\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.278673 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.278565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-secret-metrics-server-client-certs\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.278673 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.278595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-client-ca-bundle\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.278673 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.278628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-secret-metrics-server-tls\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.278673 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.278658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e6a477b5-895e-4688-ad78-cb2c0ee2422e-metrics-server-audit-profiles\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.278912 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.278691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6a477b5-895e-4688-ad78-cb2c0ee2422e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.321640 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.321598    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"]
Jun 02 15:47:54.324812 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.324789    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"
Jun 02 15:47:54.327186 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.327166    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-6h55w\""
Jun 02 15:47:54.327305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.327234    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jun 02 15:47:54.337011 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.336986    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"]
Jun 02 15:47:54.379287 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.379258    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6a477b5-895e-4688-ad78-cb2c0ee2422e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.379472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.379322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb3c660f-a1d0-4f45-9cfe-3c704c609c5e-monitoring-plugin-cert\") pod \"monitoring-plugin-5768fcc9dd-cdxhj\" (UID: \"cb3c660f-a1d0-4f45-9cfe-3c704c609c5e\") " pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"
Jun 02 15:47:54.379472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.379378    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e6a477b5-895e-4688-ad78-cb2c0ee2422e-audit-log\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.379472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.379440    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xsh\" (UniqueName: \"kubernetes.io/projected/e6a477b5-895e-4688-ad78-cb2c0ee2422e-kube-api-access-b8xsh\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.379472 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.379468    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-secret-metrics-server-client-certs\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.379694 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.379495    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-client-ca-bundle\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.379694 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.379528    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-secret-metrics-server-tls\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.379694 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.379563    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e6a477b5-895e-4688-ad78-cb2c0ee2422e-metrics-server-audit-profiles\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.380161 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.380116    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e6a477b5-895e-4688-ad78-cb2c0ee2422e-audit-log\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.380161 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.380136    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6a477b5-895e-4688-ad78-cb2c0ee2422e-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.380618 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.380592    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e6a477b5-895e-4688-ad78-cb2c0ee2422e-metrics-server-audit-profiles\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.382482 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.382438    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-secret-metrics-server-client-certs\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.382565 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.382528    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-secret-metrics-server-tls\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.382625 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.382596    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e6a477b5-895e-4688-ad78-cb2c0ee2422e-client-ca-bundle\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.393057 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.393029    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xsh\" (UniqueName: \"kubernetes.io/projected/e6a477b5-895e-4688-ad78-cb2c0ee2422e-kube-api-access-b8xsh\") pod \"metrics-server-67ddbc6856-ws4wh\" (UID: \"e6a477b5-895e-4688-ad78-cb2c0ee2422e\") " pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.480081 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.479999    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb3c660f-a1d0-4f45-9cfe-3c704c609c5e-monitoring-plugin-cert\") pod \"monitoring-plugin-5768fcc9dd-cdxhj\" (UID: \"cb3c660f-a1d0-4f45-9cfe-3c704c609c5e\") " pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"
Jun 02 15:47:54.480233 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:54.480158    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jun 02 15:47:54.480233 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:47:54.480226    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3c660f-a1d0-4f45-9cfe-3c704c609c5e-monitoring-plugin-cert podName:cb3c660f-a1d0-4f45-9cfe-3c704c609c5e nodeName:}" failed. No retries permitted until 2026-06-02 15:47:54.980207698 +0000 UTC m=+79.322628898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/cb3c660f-a1d0-4f45-9cfe-3c704c609c5e-monitoring-plugin-cert") pod "monitoring-plugin-5768fcc9dd-cdxhj" (UID: "cb3c660f-a1d0-4f45-9cfe-3c704c609c5e") : secret "monitoring-plugin-cert" not found
Jun 02 15:47:54.543267 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.543227    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:47:54.985476 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.985418    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb3c660f-a1d0-4f45-9cfe-3c704c609c5e-monitoring-plugin-cert\") pod \"monitoring-plugin-5768fcc9dd-cdxhj\" (UID: \"cb3c660f-a1d0-4f45-9cfe-3c704c609c5e\") " pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"
Jun 02 15:47:54.988180 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:54.988148    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cb3c660f-a1d0-4f45-9cfe-3c704c609c5e-monitoring-plugin-cert\") pod \"monitoring-plugin-5768fcc9dd-cdxhj\" (UID: \"cb3c660f-a1d0-4f45-9cfe-3c704c609c5e\") " pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"
Jun 02 15:47:55.234903 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.234779    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"
Jun 02 15:47:55.944241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.944205    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:47:55.949439 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.949417    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.954128 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954073    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jun 02 15:47:55.954257 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954079    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jun 02 15:47:55.954555 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954536    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jun 02 15:47:55.954683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954589    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6eggo35r2v6nd\""
Jun 02 15:47:55.954683 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954624    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jun 02 15:47:55.954798 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954567    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jun 02 15:47:55.954848 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954806    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jun 02 15:47:55.954848 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954843    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jun 02 15:47:55.954939 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.954808    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jun 02 15:47:55.955112 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.955088    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sqglb\""
Jun 02 15:47:55.955216 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.955136    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jun 02 15:47:55.955836 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.955817    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jun 02 15:47:55.957259 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.957238    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jun 02 15:47:55.961319 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.961294    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jun 02 15:47:55.968078 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.968054    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:47:55.993237 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993448    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993477    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config-out\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993576    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993602    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.993651 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.994114 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkbd\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-kube-api-access-cqkbd\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.994114 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.994114 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-web-config\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.994114 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.994114 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.994114 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:55.994114 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:55.993906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.094526 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.094489    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.094699 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.094554    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.094699 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.094589    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.094802 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.094720    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.094802 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.094758    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.095228 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.095080    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.095228 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.095133    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.095228 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.095178    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config-out\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.095481 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.095415    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096413    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096425    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096466    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096500    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096533    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096560    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkbd\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-kube-api-access-cqkbd\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096598    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096623    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-web-config\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096656    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096687    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.096737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.096713    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.097432 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.097409    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.097936 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.097911    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config-out\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.098104 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.098085    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.099004 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.098323    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.099004 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.098363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.099004 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.098520    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.099004 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.098659    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.099265 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.099030    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.099326 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.099301    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.100361 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.100307    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.101133 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.101092    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.101278 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.101253    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-web-config\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.101421 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.101375    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.101529 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.101511    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.101616 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.101595    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.108757 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.108737    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkbd\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-kube-api-access-cqkbd\") pod \"prometheus-k8s-0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.262518 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.262438    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:47:56.535116 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:47:56.535017    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jllqh"
Jun 02 15:48:00.503427 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.503379    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67ddbc6856-ws4wh"]
Jun 02 15:48:00.505794 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:48:00.505769    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a477b5_895e_4688_ad78_cb2c0ee2422e.slice/crio-9453ca33f8ad19e7336e3ea9d9470b06839c1ef02ed223baa2c7ad0c5d7db167 WatchSource:0}: Error finding container 9453ca33f8ad19e7336e3ea9d9470b06839c1ef02ed223baa2c7ad0c5d7db167: Status 404 returned error can't find the container with id 9453ca33f8ad19e7336e3ea9d9470b06839c1ef02ed223baa2c7ad0c5d7db167
Jun 02 15:48:00.596044 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.595999    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh" event={"ID":"e6a477b5-895e-4688-ad78-cb2c0ee2422e","Type":"ContainerStarted","Data":"9453ca33f8ad19e7336e3ea9d9470b06839c1ef02ed223baa2c7ad0c5d7db167"}
Jun 02 15:48:00.597711 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.597677    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-c65b8dc4b-l8vns" event={"ID":"3fc9bebc-7481-4323-9202-4f0a72ac4151","Type":"ContainerStarted","Data":"b5cf219fb0df10bfbb3b716b0c36ca95c131ee6bb69cc29606729cc680fd13c2"}
Jun 02 15:48:00.598081 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.598038    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-c65b8dc4b-l8vns"
Jun 02 15:48:00.599545 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.599499    2577 patch_prober.go:28] interesting pod/downloads-c65b8dc4b-l8vns container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.8:8080/\": dial tcp 10.134.0.8:8080: connect: connection refused" start-of-body=
Jun 02 15:48:00.599545 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.599554    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-c65b8dc4b-l8vns" podUID="3fc9bebc-7481-4323-9202-4f0a72ac4151" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.8:8080/\": dial tcp 10.134.0.8:8080: connect: connection refused"
Jun 02 15:48:00.618295 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.618189    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-c65b8dc4b-l8vns" podStartSLOduration=1.133784046 podStartE2EDuration="22.618171265s" podCreationTimestamp="2026-06-02 15:47:38 +0000 UTC" firstStartedPulling="2026-06-02 15:47:38.978903463 +0000 UTC m=+63.321324677" lastFinishedPulling="2026-06-02 15:48:00.463290692 +0000 UTC m=+84.805711896" observedRunningTime="2026-06-02 15:48:00.617330902 +0000 UTC m=+84.959752125" watchObservedRunningTime="2026-06-02 15:48:00.618171265 +0000 UTC m=+84.960592512"
Jun 02 15:48:00.733317 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.733252    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"]
Jun 02 15:48:00.737737 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:48:00.737694    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3c660f_a1d0_4f45_9cfe_3c704c609c5e.slice/crio-18c6a002e6a41a667508ca010f75235ade6645ccfc94c540855be69bbd8bd480 WatchSource:0}: Error finding container 18c6a002e6a41a667508ca010f75235ade6645ccfc94c540855be69bbd8bd480: Status 404 returned error can't find the container with id 18c6a002e6a41a667508ca010f75235ade6645ccfc94c540855be69bbd8bd480
Jun 02 15:48:00.745797 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:00.744828    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:48:00.751110 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:48:00.751070    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb5f110_9389_4545_b0a7_b6f0a7d79aa0.slice/crio-906e45e76ab262ee0b5ecb98e3bc492a7ff35cd8dc31e3d5596c3ac911af3c3e WatchSource:0}: Error finding container 906e45e76ab262ee0b5ecb98e3bc492a7ff35cd8dc31e3d5596c3ac911af3c3e: Status 404 returned error can't find the container with id 906e45e76ab262ee0b5ecb98e3bc492a7ff35cd8dc31e3d5596c3ac911af3c3e
Jun 02 15:48:01.604725 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:01.604640    2577 generic.go:358] "Generic (PLEG): container finished" podID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb" exitCode=0
Jun 02 15:48:01.605189 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:01.604743    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerDied","Data":"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"}
Jun 02 15:48:01.605189 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:01.604772    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerStarted","Data":"906e45e76ab262ee0b5ecb98e3bc492a7ff35cd8dc31e3d5596c3ac911af3c3e"}
Jun 02 15:48:01.608661 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:01.608513    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj" event={"ID":"cb3c660f-a1d0-4f45-9cfe-3c704c609c5e","Type":"ContainerStarted","Data":"18c6a002e6a41a667508ca010f75235ade6645ccfc94c540855be69bbd8bd480"}
Jun 02 15:48:01.622172 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:01.622137    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-c65b8dc4b-l8vns"
Jun 02 15:48:04.624875 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.624796    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerStarted","Data":"0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d"}
Jun 02 15:48:04.625379 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.625356    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerStarted","Data":"9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c"}
Jun 02 15:48:04.625508 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.625494    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerStarted","Data":"0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8"}
Jun 02 15:48:04.625612 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.625597    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerStarted","Data":"d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69"}
Jun 02 15:48:04.625694 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.625681    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerStarted","Data":"67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c"}
Jun 02 15:48:04.627485 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.627455    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj" event={"ID":"cb3c660f-a1d0-4f45-9cfe-3c704c609c5e","Type":"ContainerStarted","Data":"3cc239098b6d4f24841cfdcf3ef9951798fec15c6c92e33b5a729ad767937d5c"}
Jun 02 15:48:04.628423 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.627638    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"
Jun 02 15:48:04.629950 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.629919    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh" event={"ID":"e6a477b5-895e-4688-ad78-cb2c0ee2422e","Type":"ContainerStarted","Data":"9e91dcf2262bed645b245457b38ec615d5e15f0273b0262779011794c22af349"}
Jun 02 15:48:04.633820 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.633796    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj"
Jun 02 15:48:04.647455 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.647375    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5768fcc9dd-cdxhj" podStartSLOduration=7.636241268 podStartE2EDuration="10.64735906s" podCreationTimestamp="2026-06-02 15:47:54 +0000 UTC" firstStartedPulling="2026-06-02 15:48:00.740137036 +0000 UTC m=+85.082558237" lastFinishedPulling="2026-06-02 15:48:03.751254819 +0000 UTC m=+88.093676029" observedRunningTime="2026-06-02 15:48:04.645743689 +0000 UTC m=+88.988164912" watchObservedRunningTime="2026-06-02 15:48:04.64735906 +0000 UTC m=+88.989780284"
Jun 02 15:48:04.667779 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:04.667730    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh" podStartSLOduration=7.4284078319999995 podStartE2EDuration="10.66771415s" podCreationTimestamp="2026-06-02 15:47:54 +0000 UTC" firstStartedPulling="2026-06-02 15:48:00.508103027 +0000 UTC m=+84.850524227" lastFinishedPulling="2026-06-02 15:48:03.747409342 +0000 UTC m=+88.089830545" observedRunningTime="2026-06-02 15:48:04.666267051 +0000 UTC m=+89.008688273" watchObservedRunningTime="2026-06-02 15:48:04.66771415 +0000 UTC m=+89.010135385"
Jun 02 15:48:06.639536 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:06.639491    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerStarted","Data":"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"}
Jun 02 15:48:06.639536 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:06.639533    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerStarted","Data":"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"}
Jun 02 15:48:07.655463 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:07.653945    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerStarted","Data":"1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48"}
Jun 02 15:48:07.699295 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:07.695638    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.534488922 podStartE2EDuration="17.695615625s" podCreationTimestamp="2026-06-02 15:47:50 +0000 UTC" firstStartedPulling="2026-06-02 15:47:51.807482028 +0000 UTC m=+76.149903232" lastFinishedPulling="2026-06-02 15:48:06.968608694 +0000 UTC m=+91.311029935" observedRunningTime="2026-06-02 15:48:07.690127087 +0000 UTC m=+92.032548324" watchObservedRunningTime="2026-06-02 15:48:07.695615625 +0000 UTC m=+92.038036847"
Jun 02 15:48:09.662499 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:09.662469    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerStarted","Data":"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"}
Jun 02 15:48:09.662816 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:09.662502    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerStarted","Data":"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"}
Jun 02 15:48:10.668106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:10.668071    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerStarted","Data":"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"}
Jun 02 15:48:10.668106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:10.668106    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerStarted","Data":"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"}
Jun 02 15:48:11.263237 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:11.263196    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:48:14.544259 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:14.544225    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:48:14.544653 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:14.544288    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:48:17.535326 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:17.535293    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wp2zq"
Jun 02 15:48:17.567058 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:17.566990    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=14.801277141 podStartE2EDuration="22.566970396s" podCreationTimestamp="2026-06-02 15:47:55 +0000 UTC" firstStartedPulling="2026-06-02 15:48:01.606496094 +0000 UTC m=+85.948917293" lastFinishedPulling="2026-06-02 15:48:09.372189338 +0000 UTC m=+93.714610548" observedRunningTime="2026-06-02 15:48:10.712840036 +0000 UTC m=+95.055261285" watchObservedRunningTime="2026-06-02 15:48:17.566970396 +0000 UTC m=+101.909391626"
Jun 02 15:48:34.548304 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:34.548268    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:48:34.552414 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:34.552372    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67ddbc6856-ws4wh"
Jun 02 15:48:56.262812 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:56.262779    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:48:56.281302 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:56.281274    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:48:56.816084 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:48:56.816058    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:10.074034 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.073997    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:10.074680 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.074622    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="alertmanager" containerID="cri-o://67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c" gracePeriod=120
Jun 02 15:49:10.074835 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.074681    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy-metric" containerID="cri-o://0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d" gracePeriod=120
Jun 02 15:49:10.074835 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.074712    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="prom-label-proxy" containerID="cri-o://1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48" gracePeriod=120
Jun 02 15:49:10.074835 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.074690    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy-web" containerID="cri-o://0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8" gracePeriod=120
Jun 02 15:49:10.074835 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.074744    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="config-reloader" containerID="cri-o://d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69" gracePeriod=120
Jun 02 15:49:10.075045 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.074800    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy" containerID="cri-o://9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c" gracePeriod=120
Jun 02 15:49:10.843833 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843800    2577 generic.go:358] "Generic (PLEG): container finished" podID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerID="1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48" exitCode=0
Jun 02 15:49:10.843833 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843828    2577 generic.go:358] "Generic (PLEG): container finished" podID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerID="0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d" exitCode=0
Jun 02 15:49:10.843833 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843836    2577 generic.go:358] "Generic (PLEG): container finished" podID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerID="9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c" exitCode=0
Jun 02 15:49:10.844070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843844    2577 generic.go:358] "Generic (PLEG): container finished" podID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerID="d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69" exitCode=0
Jun 02 15:49:10.844070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843853    2577 generic.go:358] "Generic (PLEG): container finished" podID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerID="67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c" exitCode=0
Jun 02 15:49:10.844070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843868    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerDied","Data":"1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48"}
Jun 02 15:49:10.844070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843899    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerDied","Data":"0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d"}
Jun 02 15:49:10.844070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843911    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerDied","Data":"9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c"}
Jun 02 15:49:10.844070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843922    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerDied","Data":"d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69"}
Jun 02 15:49:10.844070 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:10.843934    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerDied","Data":"67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c"}
Jun 02 15:49:11.315502 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.315479    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:11.450725 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.450688    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-volume\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.450725 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.450734    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.450993 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.450754    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-out\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.450993 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.450777    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-metrics-client-ca\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.450993 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.450933    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-tls-assets\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451154 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451003    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451154 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451036    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k98fl\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-kube-api-access-k98fl\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451154 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451079    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-web-config\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451157    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-main-db\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451169    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:11.451305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451208    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-web\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451237    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-cluster-tls-config\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451267    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451292    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-trusted-ca-bundle\") pod \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\" (UID: \"0252993b-dcf8-44cb-92ca-4c25ab4eeb06\") "
Jun 02 15:49:11.451624 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451563    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-metrics-client-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.452236 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.451951    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:11.452236 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.452025    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:49:11.453918 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.453831    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:11.453918 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.453911    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-out" (OuterVolumeSpecName: "config-out") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:49:11.454092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.453978    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-volume" (OuterVolumeSpecName: "config-volume") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:11.454330 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.454290    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:49:11.454652 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.454619    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:11.454848 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.454813    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:11.454950 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.454912    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:11.455998 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.455977    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-kube-api-access-k98fl" (OuterVolumeSpecName: "kube-api-access-k98fl") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "kube-api-access-k98fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:49:11.459078 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.459055    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:11.465290 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.465267    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-web-config" (OuterVolumeSpecName: "web-config") pod "0252993b-dcf8-44cb-92ca-4c25ab4eeb06" (UID: "0252993b-dcf8-44cb-92ca-4c25ab4eeb06"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:11.552714 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552679    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98fl\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-kube-api-access-k98fl\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552714 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552708    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-web-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552714 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552718    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-main-db\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552730    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552740    2577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-cluster-tls-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552750    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552759    2577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552767    2577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-volume\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552776    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552785    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-config-out\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552793    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-tls-assets\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.552942 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.552801    2577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0252993b-dcf8-44cb-92ca-4c25ab4eeb06-secret-alertmanager-main-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:11.849112 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.849019    2577 generic.go:358] "Generic (PLEG): container finished" podID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerID="0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8" exitCode=0
Jun 02 15:49:11.849112 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.849097    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerDied","Data":"0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8"}
Jun 02 15:49:11.849375 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.849132    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:11.849375 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.849146    2577 scope.go:117] "RemoveContainer" containerID="1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48"
Jun 02 15:49:11.849375 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.849136    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0252993b-dcf8-44cb-92ca-4c25ab4eeb06","Type":"ContainerDied","Data":"bf6776558bceffe0de239d06270efe5eb14fc9a1dc6f1ef7af63fc6a64916428"}
Jun 02 15:49:11.856863 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.856844    2577 scope.go:117] "RemoveContainer" containerID="0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d"
Jun 02 15:49:11.864194 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.864178    2577 scope.go:117] "RemoveContainer" containerID="9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c"
Jun 02 15:49:11.870490 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.870472    2577 scope.go:117] "RemoveContainer" containerID="0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8"
Jun 02 15:49:11.877831 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.877583    2577 scope.go:117] "RemoveContainer" containerID="d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69"
Jun 02 15:49:11.877831 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.877648    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:11.884091 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.884076    2577 scope.go:117] "RemoveContainer" containerID="67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c"
Jun 02 15:49:11.887618 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.887596    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:11.891169 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.891152    2577 scope.go:117] "RemoveContainer" containerID="8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078"
Jun 02 15:49:11.897190 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.897172    2577 scope.go:117] "RemoveContainer" containerID="1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48"
Jun 02 15:49:11.897514 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:11.897493    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48\": container with ID starting with 1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48 not found: ID does not exist" containerID="1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48"
Jun 02 15:49:11.897587 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.897522    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48"} err="failed to get container status \"1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48\": rpc error: code = NotFound desc = could not find container \"1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48\": container with ID starting with 1526da6a0f0abf983ca78edb351982cd7d902d04566aa7fffbdc5e30e15caf48 not found: ID does not exist"
Jun 02 15:49:11.897587 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.897557    2577 scope.go:117] "RemoveContainer" containerID="0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d"
Jun 02 15:49:11.897801 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:11.897784    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d\": container with ID starting with 0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d not found: ID does not exist" containerID="0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d"
Jun 02 15:49:11.897851 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.897804    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d"} err="failed to get container status \"0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d\": rpc error: code = NotFound desc = could not find container \"0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d\": container with ID starting with 0c5a85cd3d31dac3c422f26087b1398de757f734eee4b46e902bcae8e8fe592d not found: ID does not exist"
Jun 02 15:49:11.897851 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.897818    2577 scope.go:117] "RemoveContainer" containerID="9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c"
Jun 02 15:49:11.898042 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:11.898026    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c\": container with ID starting with 9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c not found: ID does not exist" containerID="9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c"
Jun 02 15:49:11.898087 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.898049    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c"} err="failed to get container status \"9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c\": rpc error: code = NotFound desc = could not find container \"9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c\": container with ID starting with 9f3e8607129827d738f354f4d72dbe3a867054110cb3e3b230eecdb0b1198a2c not found: ID does not exist"
Jun 02 15:49:11.898087 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.898075    2577 scope.go:117] "RemoveContainer" containerID="0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8"
Jun 02 15:49:11.898296 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:11.898281    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8\": container with ID starting with 0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8 not found: ID does not exist" containerID="0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8"
Jun 02 15:49:11.898338 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.898301    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8"} err="failed to get container status \"0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8\": rpc error: code = NotFound desc = could not find container \"0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8\": container with ID starting with 0ded46cba426397d8c2c143098fbcb03b6c0f0fbf9b6781e22b2a6c010fda2f8 not found: ID does not exist"
Jun 02 15:49:11.898338 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.898314    2577 scope.go:117] "RemoveContainer" containerID="d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69"
Jun 02 15:49:11.898568 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:11.898552    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69\": container with ID starting with d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69 not found: ID does not exist" containerID="d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69"
Jun 02 15:49:11.898611 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.898573    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69"} err="failed to get container status \"d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69\": rpc error: code = NotFound desc = could not find container \"d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69\": container with ID starting with d79f554e320eddcad5c982602ad4cd9ab8330262bb18d269a64915e2b5713b69 not found: ID does not exist"
Jun 02 15:49:11.898611 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.898588    2577 scope.go:117] "RemoveContainer" containerID="67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c"
Jun 02 15:49:11.898802 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:11.898788    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c\": container with ID starting with 67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c not found: ID does not exist" containerID="67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c"
Jun 02 15:49:11.898843 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.898806    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c"} err="failed to get container status \"67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c\": rpc error: code = NotFound desc = could not find container \"67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c\": container with ID starting with 67d5d815dd626c7c1fa1132a73570aaceef2496da84d10672ad91c8961c8775c not found: ID does not exist"
Jun 02 15:49:11.898843 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.898819    2577 scope.go:117] "RemoveContainer" containerID="8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078"
Jun 02 15:49:11.899040 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:11.899025    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078\": container with ID starting with 8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078 not found: ID does not exist" containerID="8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078"
Jun 02 15:49:11.899082 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.899043    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078"} err="failed to get container status \"8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078\": rpc error: code = NotFound desc = could not find container \"8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078\": container with ID starting with 8de9e6c290e04f192866f2dd5759c45bb38e80b926621b651ee9a8feaf88d078 not found: ID does not exist"
Jun 02 15:49:11.922229 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922203    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:11.922555 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="prom-label-proxy"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922557    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="prom-label-proxy"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922567    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy-web"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922574    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy-web"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922587    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy-metric"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922594    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy-metric"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922602    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922607    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922617    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="config-reloader"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922622    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="config-reloader"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922631    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="init-config-reloader"
Jun 02 15:49:11.922633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922636    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="init-config-reloader"
Jun 02 15:49:11.922955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922643    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="alertmanager"
Jun 02 15:49:11.922955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922649    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="alertmanager"
Jun 02 15:49:11.922955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922695    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="config-reloader"
Jun 02 15:49:11.922955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922704    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy-web"
Jun 02 15:49:11.922955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922710    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy-metric"
Jun 02 15:49:11.922955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922716    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="prom-label-proxy"
Jun 02 15:49:11.922955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922724    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="kube-rbac-proxy"
Jun 02 15:49:11.922955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.922729    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" containerName="alertmanager"
Jun 02 15:49:11.926257 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.926241    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:11.928445 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.928417    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jun 02 15:49:11.928538 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.928444    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jun 02 15:49:11.928538 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.928445    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jun 02 15:49:11.928650 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.928601    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jun 02 15:49:11.928650 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.928639    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jun 02 15:49:11.928766 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.928751    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jun 02 15:49:11.928852 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.928839    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jun 02 15:49:11.929197 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.929183    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cd5f5\""
Jun 02 15:49:11.929325 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.929310    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jun 02 15:49:11.934181 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.934163    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jun 02 15:49:11.942213 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:11.942188    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:12.055976 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.055935    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.055976 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.055980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056183 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056017    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056183 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056183 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056118    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056183 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-web-config\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056183 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056185    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl2gp\" (UniqueName: \"kubernetes.io/projected/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-kube-api-access-xl2gp\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-config-out\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-config-volume\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.056521 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.056349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157034 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.156994    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-web-config\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157034 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157037    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157066    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xl2gp\" (UniqueName: \"kubernetes.io/projected/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-kube-api-access-xl2gp\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157097    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157120    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-config-out\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157136    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157177    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-config-volume\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157200    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157241 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157241    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157670 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157270    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157670 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157308    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157670 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157330    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.157670 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.157371    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.158159 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.158131    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.158437 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.158414    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.158592 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.158569    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.160237 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.160157    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-config-out\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.160237 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.160184    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.160432 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.160238    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.160432 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.160279    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-config-volume\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.160432 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.160363    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.160912 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.160887    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.161051 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.161033    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.161120 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.161058    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.162133 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.162115    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-web-config\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.168931 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.168910    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl2gp\" (UniqueName: \"kubernetes.io/projected/e4c2e8b9-46e7-4320-99a0-89a78f98ed27-kube-api-access-xl2gp\") pod \"alertmanager-main-0\" (UID: \"e4c2e8b9-46e7-4320-99a0-89a78f98ed27\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.222826 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.222793    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0252993b-dcf8-44cb-92ca-4c25ab4eeb06" path="/var/lib/kubelet/pods/0252993b-dcf8-44cb-92ca-4c25ab4eeb06/volumes"
Jun 02 15:49:12.235566 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.235534    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 02 15:49:12.379406 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.379354    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 02 15:49:12.382026 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:49:12.381997    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c2e8b9_46e7_4320_99a0_89a78f98ed27.slice/crio-57e623110a4a22dd1056d282fc5e2bef7fa60648bec3f278efe36ac181c891ca WatchSource:0}: Error finding container 57e623110a4a22dd1056d282fc5e2bef7fa60648bec3f278efe36ac181c891ca: Status 404 returned error can't find the container with id 57e623110a4a22dd1056d282fc5e2bef7fa60648bec3f278efe36ac181c891ca
Jun 02 15:49:12.853352 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.853317    2577 generic.go:358] "Generic (PLEG): container finished" podID="e4c2e8b9-46e7-4320-99a0-89a78f98ed27" containerID="462d7bb4cb881452a9163f8d116f0e690eb616cc9e0894075ad78a7040c1c4f7" exitCode=0
Jun 02 15:49:12.853563 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.853414    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4c2e8b9-46e7-4320-99a0-89a78f98ed27","Type":"ContainerDied","Data":"462d7bb4cb881452a9163f8d116f0e690eb616cc9e0894075ad78a7040c1c4f7"}
Jun 02 15:49:12.853563 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:12.853451    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4c2e8b9-46e7-4320-99a0-89a78f98ed27","Type":"ContainerStarted","Data":"57e623110a4a22dd1056d282fc5e2bef7fa60648bec3f278efe36ac181c891ca"}
Jun 02 15:49:13.860538 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:13.860501    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4c2e8b9-46e7-4320-99a0-89a78f98ed27","Type":"ContainerStarted","Data":"f13b8c4affee0e7e334838b77943b0ddb62099feffd34b2212e0a79e47a7778c"}
Jun 02 15:49:13.860538 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:13.860539    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4c2e8b9-46e7-4320-99a0-89a78f98ed27","Type":"ContainerStarted","Data":"cc81abaa36df355c8a55b41af78450f97d644221ec9410d16108d56757d872ef"}
Jun 02 15:49:13.860538 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:13.860549    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4c2e8b9-46e7-4320-99a0-89a78f98ed27","Type":"ContainerStarted","Data":"c1f0bc6cd67f854d89fb31069f60997692f1ac4e05a131f4b6ed94989fb560e7"}
Jun 02 15:49:13.860996 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:13.860558    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4c2e8b9-46e7-4320-99a0-89a78f98ed27","Type":"ContainerStarted","Data":"7e0cc3cb2eb9fc05384805aab9957c793141e3403fb5aa2d3cacc6d97a7f89c1"}
Jun 02 15:49:13.860996 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:13.860567    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4c2e8b9-46e7-4320-99a0-89a78f98ed27","Type":"ContainerStarted","Data":"312652ef2a540db6853ef544d27dde8aa2b52255d7bffa557f38e4f100fed20e"}
Jun 02 15:49:13.860996 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:13.860576    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4c2e8b9-46e7-4320-99a0-89a78f98ed27","Type":"ContainerStarted","Data":"cdb2fb1dc5d3f3c40d732bb4c3186a6e1923947737867ae18f207f0a1eaec97e"}
Jun 02 15:49:13.890353 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:13.890294    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.890279513 podStartE2EDuration="2.890279513s" podCreationTimestamp="2026-06-02 15:49:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:13.890090454 +0000 UTC m=+158.232511677" watchObservedRunningTime="2026-06-02 15:49:13.890279513 +0000 UTC m=+158.232700737"
Jun 02 15:49:14.201023 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.200986    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"]
Jun 02 15:49:14.203299 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.203284    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.205700 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.205669    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jun 02 15:49:14.205700 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.205670    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jun 02 15:49:14.205876 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.205707    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jun 02 15:49:14.205876 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.205737    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jun 02 15:49:14.205965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.205930    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jun 02 15:49:14.206018 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.206000    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-f26tg\""
Jun 02 15:49:14.211229 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.211212    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jun 02 15:49:14.223813 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.223792    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"]
Jun 02 15:49:14.376940 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.376906    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd4bw\" (UniqueName: \"kubernetes.io/projected/0e1a51e0-f52e-4d54-8d67-dad858c300de-kube-api-access-fd4bw\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.377104 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.376968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-telemeter-client-tls\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.377104 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.377052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-metrics-client-ca\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.377104 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.377083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-federate-client-tls\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.377245 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.377113    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-secret-telemeter-client\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.377245 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.377132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-serving-certs-ca-bundle\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.377245 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.377228    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.381681 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.377305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.478493 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.478381    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.478493 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.478452    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fd4bw\" (UniqueName: \"kubernetes.io/projected/0e1a51e0-f52e-4d54-8d67-dad858c300de-kube-api-access-fd4bw\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.478714 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.478500    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-telemeter-client-tls\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.478714 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.478549    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-metrics-client-ca\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.478714 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.478575    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-federate-client-tls\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.478714 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.478600    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-secret-telemeter-client\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.478911 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.478749    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-serving-certs-ca-bundle\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.478911 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.478809    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.479550 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.479520    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-serving-certs-ca-bundle\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.479670 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.479643    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-metrics-client-ca\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.479737 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.479650    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e1a51e0-f52e-4d54-8d67-dad858c300de-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.481149 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.481122    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.481323 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.481305    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-secret-telemeter-client\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.481416 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.481377    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-telemeter-client-tls\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.481462 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.481437    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0e1a51e0-f52e-4d54-8d67-dad858c300de-federate-client-tls\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.489809 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.489790    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd4bw\" (UniqueName: \"kubernetes.io/projected/0e1a51e0-f52e-4d54-8d67-dad858c300de-kube-api-access-fd4bw\") pod \"telemeter-client-6dc5d86cb8-b8vrp\" (UID: \"0e1a51e0-f52e-4d54-8d67-dad858c300de\") " pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.498519 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.498497    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:14.498959 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.498911    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="prometheus" containerID="cri-o://d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3" gracePeriod=600
Jun 02 15:49:14.499092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.498947    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="config-reloader" containerID="cri-o://5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763" gracePeriod=600
Jun 02 15:49:14.499092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.498949    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f" gracePeriod=600
Jun 02 15:49:14.499092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.498963    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy-web" containerID="cri-o://5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1" gracePeriod=600
Jun 02 15:49:14.499092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.498915    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy" containerID="cri-o://91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e" gracePeriod=600
Jun 02 15:49:14.499423 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.498943    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="thanos-sidecar" containerID="cri-o://5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed" gracePeriod=600
Jun 02 15:49:14.512322 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.512297    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"
Jun 02 15:49:14.667971 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.667932    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp"]
Jun 02 15:49:14.670963 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:49:14.670934    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1a51e0_f52e_4d54_8d67_dad858c300de.slice/crio-78d3079c862ca4d01c471d1fddbfeb7d6ac56f77b7556031c041a1b0bac011ed WatchSource:0}: Error finding container 78d3079c862ca4d01c471d1fddbfeb7d6ac56f77b7556031c041a1b0bac011ed: Status 404 returned error can't find the container with id 78d3079c862ca4d01c471d1fddbfeb7d6ac56f77b7556031c041a1b0bac011ed
Jun 02 15:49:14.736199 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.736175    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:14.866990 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.866948    2577 generic.go:358] "Generic (PLEG): container finished" podID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f" exitCode=0
Jun 02 15:49:14.866990 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.866977    2577 generic.go:358] "Generic (PLEG): container finished" podID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e" exitCode=0
Jun 02 15:49:14.866990 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.866993    2577 generic.go:358] "Generic (PLEG): container finished" podID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1" exitCode=0
Jun 02 15:49:14.866990 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867000    2577 generic.go:358] "Generic (PLEG): container finished" podID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed" exitCode=0
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867011    2577 generic.go:358] "Generic (PLEG): container finished" podID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763" exitCode=0
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867018    2577 generic.go:358] "Generic (PLEG): container finished" podID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3" exitCode=0
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867028    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerDied","Data":"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"}
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867072    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerDied","Data":"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"}
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867079    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867091    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerDied","Data":"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"}
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867105    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerDied","Data":"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"}
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867120    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerDied","Data":"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"}
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867134    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerDied","Data":"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"}
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867140    2577 scope.go:117] "RemoveContainer" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"
Jun 02 15:49:14.867609 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.867150    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0","Type":"ContainerDied","Data":"906e45e76ab262ee0b5ecb98e3bc492a7ff35cd8dc31e3d5596c3ac911af3c3e"}
Jun 02 15:49:14.868250 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.868224    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp" event={"ID":"0e1a51e0-f52e-4d54-8d67-dad858c300de","Type":"ContainerStarted","Data":"78d3079c862ca4d01c471d1fddbfeb7d6ac56f77b7556031c041a1b0bac011ed"}
Jun 02 15:49:14.874169 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.874138    2577 scope.go:117] "RemoveContainer" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"
Jun 02 15:49:14.881012 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.880997    2577 scope.go:117] "RemoveContainer" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"
Jun 02 15:49:14.881740 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.881718    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-trusted-ca-bundle\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.881846 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.881767    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.881846 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.881803    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-db\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.881846 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.881835    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-tls-assets\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882063 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.881873    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882063 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.881910    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-kube-rbac-proxy\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882525 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882504    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-thanos-prometheus-http-client-file\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882693 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882668    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-rulefiles-0\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882817 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882706    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-metrics-client-ca\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882817 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882744    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqkbd\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-kube-api-access-cqkbd\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882817 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882785    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config-out\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882986 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882820    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-serving-certs-ca-bundle\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882986 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882848    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-tls\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882986 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882876    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-kubelet-serving-ca-bundle\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882986 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882907    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-grpc-tls\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.882986 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882936    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-web-config\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.883233 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882998    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.883233 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.883053    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-metrics-client-certs\") pod \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\" (UID: \"8cb5f110-9389-4545-b0a7-b6f0a7d79aa0\") "
Jun 02 15:49:14.884556 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882533    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:14.884556 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.882915    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:49:14.884556 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.883811    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:14.884556 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.884180    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:14.884556 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.884261    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config" (OuterVolumeSpecName: "config") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.884849 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.884738    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:14.885220 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.885157    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:49:14.885220 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.885186    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.885854 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.885792    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:49:14.885966 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.885918    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.886955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.886919    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-kube-api-access-cqkbd" (OuterVolumeSpecName: "kube-api-access-cqkbd") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "kube-api-access-cqkbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:49:14.887382 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.887336    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config-out" (OuterVolumeSpecName: "config-out") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:49:14.887552 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.887529    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.887742 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.887705    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.887814 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.887767    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.887860 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.887811    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.887939 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.887922    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.889742 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.889725    2577 scope.go:117] "RemoveContainer" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"
Jun 02 15:49:14.896656 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.896637    2577 scope.go:117] "RemoveContainer" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"
Jun 02 15:49:14.898049 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.898029    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-web-config" (OuterVolumeSpecName: "web-config") pod "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" (UID: "8cb5f110-9389-4545-b0a7-b6f0a7d79aa0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:49:14.903100 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.903086    2577 scope.go:117] "RemoveContainer" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"
Jun 02 15:49:14.909797 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.909772    2577 scope.go:117] "RemoveContainer" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"
Jun 02 15:49:14.918001 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.917977    2577 scope.go:117] "RemoveContainer" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"
Jun 02 15:49:14.918431 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:14.918409    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": container with ID starting with a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f not found: ID does not exist" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"
Jun 02 15:49:14.918510 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.918439    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"} err="failed to get container status \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": rpc error: code = NotFound desc = could not find container \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": container with ID starting with a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f not found: ID does not exist"
Jun 02 15:49:14.918510 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.918457    2577 scope.go:117] "RemoveContainer" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"
Jun 02 15:49:14.918721 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:14.918700    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": container with ID starting with 91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e not found: ID does not exist" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"
Jun 02 15:49:14.918776 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.918728    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"} err="failed to get container status \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": rpc error: code = NotFound desc = could not find container \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": container with ID starting with 91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e not found: ID does not exist"
Jun 02 15:49:14.918776 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.918746    2577 scope.go:117] "RemoveContainer" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"
Jun 02 15:49:14.918969 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:14.918949    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": container with ID starting with 5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1 not found: ID does not exist" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"
Jun 02 15:49:14.919028 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.918974    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"} err="failed to get container status \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": rpc error: code = NotFound desc = could not find container \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": container with ID starting with 5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1 not found: ID does not exist"
Jun 02 15:49:14.919028 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.918995    2577 scope.go:117] "RemoveContainer" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"
Jun 02 15:49:14.919250 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:14.919232    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": container with ID starting with 5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed not found: ID does not exist" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"
Jun 02 15:49:14.919294 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.919255    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"} err="failed to get container status \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": rpc error: code = NotFound desc = could not find container \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": container with ID starting with 5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed not found: ID does not exist"
Jun 02 15:49:14.919294 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.919274    2577 scope.go:117] "RemoveContainer" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"
Jun 02 15:49:14.919506 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:14.919491    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": container with ID starting with 5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763 not found: ID does not exist" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"
Jun 02 15:49:14.919566 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.919509    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"} err="failed to get container status \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": rpc error: code = NotFound desc = could not find container \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": container with ID starting with 5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763 not found: ID does not exist"
Jun 02 15:49:14.919566 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.919521    2577 scope.go:117] "RemoveContainer" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"
Jun 02 15:49:14.919739 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:14.919724    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": container with ID starting with d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3 not found: ID does not exist" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"
Jun 02 15:49:14.919780 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.919741    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"} err="failed to get container status \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": rpc error: code = NotFound desc = could not find container \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": container with ID starting with d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3 not found: ID does not exist"
Jun 02 15:49:14.919780 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.919753    2577 scope.go:117] "RemoveContainer" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"
Jun 02 15:49:14.919985 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:49:14.919965    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": container with ID starting with 73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb not found: ID does not exist" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"
Jun 02 15:49:14.920023 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.919990    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"} err="failed to get container status \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": rpc error: code = NotFound desc = could not find container \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": container with ID starting with 73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb not found: ID does not exist"
Jun 02 15:49:14.920023 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920005    2577 scope.go:117] "RemoveContainer" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"
Jun 02 15:49:14.920206 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920189    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"} err="failed to get container status \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": rpc error: code = NotFound desc = could not find container \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": container with ID starting with a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f not found: ID does not exist"
Jun 02 15:49:14.920253 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920206    2577 scope.go:117] "RemoveContainer" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"
Jun 02 15:49:14.920411 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920376    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"} err="failed to get container status \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": rpc error: code = NotFound desc = could not find container \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": container with ID starting with 91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e not found: ID does not exist"
Jun 02 15:49:14.920411 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920406    2577 scope.go:117] "RemoveContainer" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"
Jun 02 15:49:14.920638 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920621    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"} err="failed to get container status \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": rpc error: code = NotFound desc = could not find container \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": container with ID starting with 5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1 not found: ID does not exist"
Jun 02 15:49:14.920676 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920638    2577 scope.go:117] "RemoveContainer" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"
Jun 02 15:49:14.920824 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920808    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"} err="failed to get container status \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": rpc error: code = NotFound desc = could not find container \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": container with ID starting with 5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed not found: ID does not exist"
Jun 02 15:49:14.920860 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920824    2577 scope.go:117] "RemoveContainer" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"
Jun 02 15:49:14.920998 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920982    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"} err="failed to get container status \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": rpc error: code = NotFound desc = could not find container \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": container with ID starting with 5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763 not found: ID does not exist"
Jun 02 15:49:14.921039 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.920999    2577 scope.go:117] "RemoveContainer" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"
Jun 02 15:49:14.921176 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921157    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"} err="failed to get container status \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": rpc error: code = NotFound desc = could not find container \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": container with ID starting with d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3 not found: ID does not exist"
Jun 02 15:49:14.921229 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921176    2577 scope.go:117] "RemoveContainer" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"
Jun 02 15:49:14.921376 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921360    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"} err="failed to get container status \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": rpc error: code = NotFound desc = could not find container \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": container with ID starting with 73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb not found: ID does not exist"
Jun 02 15:49:14.921445 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921376    2577 scope.go:117] "RemoveContainer" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"
Jun 02 15:49:14.921599 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921582    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"} err="failed to get container status \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": rpc error: code = NotFound desc = could not find container \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": container with ID starting with a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f not found: ID does not exist"
Jun 02 15:49:14.921633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921601    2577 scope.go:117] "RemoveContainer" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"
Jun 02 15:49:14.921811 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921795    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"} err="failed to get container status \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": rpc error: code = NotFound desc = could not find container \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": container with ID starting with 91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e not found: ID does not exist"
Jun 02 15:49:14.921854 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921811    2577 scope.go:117] "RemoveContainer" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"
Jun 02 15:49:14.922011 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.921993    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"} err="failed to get container status \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": rpc error: code = NotFound desc = could not find container \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": container with ID starting with 5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1 not found: ID does not exist"
Jun 02 15:49:14.922056 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922011    2577 scope.go:117] "RemoveContainer" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"
Jun 02 15:49:14.922181 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922167    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"} err="failed to get container status \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": rpc error: code = NotFound desc = could not find container \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": container with ID starting with 5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed not found: ID does not exist"
Jun 02 15:49:14.922223 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922180    2577 scope.go:117] "RemoveContainer" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"
Jun 02 15:49:14.922338 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922324    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"} err="failed to get container status \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": rpc error: code = NotFound desc = could not find container \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": container with ID starting with 5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763 not found: ID does not exist"
Jun 02 15:49:14.922338 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922337    2577 scope.go:117] "RemoveContainer" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"
Jun 02 15:49:14.922531 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922516    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"} err="failed to get container status \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": rpc error: code = NotFound desc = could not find container \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": container with ID starting with d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3 not found: ID does not exist"
Jun 02 15:49:14.922571 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922532    2577 scope.go:117] "RemoveContainer" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"
Jun 02 15:49:14.922722 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922688    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"} err="failed to get container status \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": rpc error: code = NotFound desc = could not find container \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": container with ID starting with 73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb not found: ID does not exist"
Jun 02 15:49:14.922754 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922723    2577 scope.go:117] "RemoveContainer" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"
Jun 02 15:49:14.922932 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922913    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"} err="failed to get container status \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": rpc error: code = NotFound desc = could not find container \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": container with ID starting with a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f not found: ID does not exist"
Jun 02 15:49:14.922965 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.922935    2577 scope.go:117] "RemoveContainer" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"
Jun 02 15:49:14.923122 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923099    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"} err="failed to get container status \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": rpc error: code = NotFound desc = could not find container \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": container with ID starting with 91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e not found: ID does not exist"
Jun 02 15:49:14.923122 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923116    2577 scope.go:117] "RemoveContainer" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"
Jun 02 15:49:14.923296 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923269    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"} err="failed to get container status \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": rpc error: code = NotFound desc = could not find container \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": container with ID starting with 5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1 not found: ID does not exist"
Jun 02 15:49:14.923338 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923297    2577 scope.go:117] "RemoveContainer" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"
Jun 02 15:49:14.923471 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923456    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"} err="failed to get container status \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": rpc error: code = NotFound desc = could not find container \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": container with ID starting with 5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed not found: ID does not exist"
Jun 02 15:49:14.923515 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923470    2577 scope.go:117] "RemoveContainer" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"
Jun 02 15:49:14.923643 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923624    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"} err="failed to get container status \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": rpc error: code = NotFound desc = could not find container \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": container with ID starting with 5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763 not found: ID does not exist"
Jun 02 15:49:14.923700 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923647    2577 scope.go:117] "RemoveContainer" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"
Jun 02 15:49:14.923870 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923850    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"} err="failed to get container status \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": rpc error: code = NotFound desc = could not find container \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": container with ID starting with d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3 not found: ID does not exist"
Jun 02 15:49:14.923927 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.923872    2577 scope.go:117] "RemoveContainer" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"
Jun 02 15:49:14.924055 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924039    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"} err="failed to get container status \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": rpc error: code = NotFound desc = could not find container \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": container with ID starting with 73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb not found: ID does not exist"
Jun 02 15:49:14.924111 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924058    2577 scope.go:117] "RemoveContainer" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"
Jun 02 15:49:14.924248 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924233    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"} err="failed to get container status \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": rpc error: code = NotFound desc = could not find container \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": container with ID starting with a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f not found: ID does not exist"
Jun 02 15:49:14.924306 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924250    2577 scope.go:117] "RemoveContainer" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"
Jun 02 15:49:14.924453 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924438    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"} err="failed to get container status \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": rpc error: code = NotFound desc = could not find container \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": container with ID starting with 91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e not found: ID does not exist"
Jun 02 15:49:14.924513 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924454    2577 scope.go:117] "RemoveContainer" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"
Jun 02 15:49:14.924659 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924644    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"} err="failed to get container status \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": rpc error: code = NotFound desc = could not find container \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": container with ID starting with 5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1 not found: ID does not exist"
Jun 02 15:49:14.924726 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924660    2577 scope.go:117] "RemoveContainer" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"
Jun 02 15:49:14.924854 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924839    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"} err="failed to get container status \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": rpc error: code = NotFound desc = could not find container \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": container with ID starting with 5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed not found: ID does not exist"
Jun 02 15:49:14.924910 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.924856    2577 scope.go:117] "RemoveContainer" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"
Jun 02 15:49:14.925050 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925034    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"} err="failed to get container status \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": rpc error: code = NotFound desc = could not find container \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": container with ID starting with 5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763 not found: ID does not exist"
Jun 02 15:49:14.925092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925052    2577 scope.go:117] "RemoveContainer" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"
Jun 02 15:49:14.925207 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925190    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"} err="failed to get container status \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": rpc error: code = NotFound desc = could not find container \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": container with ID starting with d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3 not found: ID does not exist"
Jun 02 15:49:14.925258 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925209    2577 scope.go:117] "RemoveContainer" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"
Jun 02 15:49:14.925421 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925402    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"} err="failed to get container status \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": rpc error: code = NotFound desc = could not find container \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": container with ID starting with 73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb not found: ID does not exist"
Jun 02 15:49:14.925464 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925423    2577 scope.go:117] "RemoveContainer" containerID="a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"
Jun 02 15:49:14.925619 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925602    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f"} err="failed to get container status \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": rpc error: code = NotFound desc = could not find container \"a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f\": container with ID starting with a8ed80d28db84737eab87c40a84aeb38aecd5b8846d166e9d14e702b15a8b59f not found: ID does not exist"
Jun 02 15:49:14.925677 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925620    2577 scope.go:117] "RemoveContainer" containerID="91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"
Jun 02 15:49:14.925821 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925805    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e"} err="failed to get container status \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": rpc error: code = NotFound desc = could not find container \"91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e\": container with ID starting with 91f8742a2884c13b10c099b872ac380ca40d27b90687896bd3f1f0be634fdc1e not found: ID does not exist"
Jun 02 15:49:14.925873 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925823    2577 scope.go:117] "RemoveContainer" containerID="5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"
Jun 02 15:49:14.926006 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.925989    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1"} err="failed to get container status \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": rpc error: code = NotFound desc = could not find container \"5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1\": container with ID starting with 5f33755857d639745b366457c4f6eb90e72e60b5316319e6a78fbd1cff7e23e1 not found: ID does not exist"
Jun 02 15:49:14.926062 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.926007    2577 scope.go:117] "RemoveContainer" containerID="5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"
Jun 02 15:49:14.926171 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.926155    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed"} err="failed to get container status \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": rpc error: code = NotFound desc = could not find container \"5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed\": container with ID starting with 5d141378bec6083fe27ecfca827063bcba5169517fa00d445a0181b96d7b19ed not found: ID does not exist"
Jun 02 15:49:14.926226 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.926172    2577 scope.go:117] "RemoveContainer" containerID="5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"
Jun 02 15:49:14.926342 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.926325    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763"} err="failed to get container status \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": rpc error: code = NotFound desc = could not find container \"5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763\": container with ID starting with 5f377ae01fa934ca342c53a495d880412009781a1899a205d46dfa8ec107a763 not found: ID does not exist"
Jun 02 15:49:14.926383 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.926343    2577 scope.go:117] "RemoveContainer" containerID="d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"
Jun 02 15:49:14.926570 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.926552    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3"} err="failed to get container status \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": rpc error: code = NotFound desc = could not find container \"d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3\": container with ID starting with d2fd6b7a19acd548839c41997c9b166a96250dc7a2e9d8bff7088e0d20d5d2b3 not found: ID does not exist"
Jun 02 15:49:14.926629 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.926571    2577 scope.go:117] "RemoveContainer" containerID="73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"
Jun 02 15:49:14.926776 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.926761    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb"} err="failed to get container status \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": rpc error: code = NotFound desc = could not find container \"73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb\": container with ID starting with 73a242b4c4eef0de71ed1eff18333f3f1d45f1fd3bbafc1dc5e7a6eb443d07fb not found: ID does not exist"
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984412    2577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984443    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-db\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984458    2577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-tls-assets\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984475    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984490    2577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-kube-rbac-proxy\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984507    2577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984523    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984540    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-metrics-client-ca\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984554    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqkbd\" (UniqueName: \"kubernetes.io/projected/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-kube-api-access-cqkbd\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984570    2577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-config-out\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984584    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984614    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984644    2577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984662    2577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-grpc-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984678    2577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-web-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984695    2577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984709    2577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-secret-metrics-client-certs\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:14.984764 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:14.984724    2577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:49:15.197625 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.197595    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:15.204298 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.204272    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:15.235423 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235343    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:15.235671 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235659    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy"
Jun 02 15:49:15.235708 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235672    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy"
Jun 02 15:49:15.235708 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235681    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="config-reloader"
Jun 02 15:49:15.235708 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235686    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="config-reloader"
Jun 02 15:49:15.235708 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235696    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="thanos-sidecar"
Jun 02 15:49:15.235708 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235701    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="thanos-sidecar"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235711    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy-web"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235717    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy-web"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235724    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="prometheus"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235729    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="prometheus"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235735    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy-thanos"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235740    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy-thanos"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235745    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="init-config-reloader"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235750    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="init-config-reloader"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235811    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy-web"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235820    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="config-reloader"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235826    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy-thanos"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235831    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="prometheus"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235837    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="thanos-sidecar"
Jun 02 15:49:15.235847 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.235845    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" containerName="kube-rbac-proxy"
Jun 02 15:49:15.237936 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.237922    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.240256 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240232    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jun 02 15:49:15.240375 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240315    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jun 02 15:49:15.240453 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240417    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jun 02 15:49:15.240453 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240424    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-sqglb\""
Jun 02 15:49:15.240561 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240467    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jun 02 15:49:15.240657 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240636    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jun 02 15:49:15.240783 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240705    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jun 02 15:49:15.240783 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240736    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jun 02 15:49:15.240783 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240778    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6eggo35r2v6nd\""
Jun 02 15:49:15.240923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240798    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jun 02 15:49:15.240923 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.240848    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jun 02 15:49:15.241174 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.241154    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jun 02 15:49:15.244194 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.244178    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jun 02 15:49:15.246604 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.246587    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jun 02 15:49:15.255977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.255952    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:15.389124 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389296 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389296 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389296 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389296 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389248    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9898e745-053c-4cdf-89b8-a0a93dad3b8d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389296 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389287    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389335    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9898e745-053c-4cdf-89b8-a0a93dad3b8d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389359    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389481    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jld5c\" (UniqueName: \"kubernetes.io/projected/9898e745-053c-4cdf-89b8-a0a93dad3b8d-kube-api-access-jld5c\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389581    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.389685 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.390006 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.390006 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-config\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.390006 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.389795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.490784 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.490729    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.490784 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.490782    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491036 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.490809    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491036 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.490835    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9898e745-053c-4cdf-89b8-a0a93dad3b8d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491036 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.490860    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491036 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.490895    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9898e745-053c-4cdf-89b8-a0a93dad3b8d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491036 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.490921    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491036 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.490962    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491036 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491002    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jld5c\" (UniqueName: \"kubernetes.io/projected/9898e745-053c-4cdf-89b8-a0a93dad3b8d-kube-api-access-jld5c\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491055    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491095    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491122    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491152    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491176    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491203    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491233    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-config\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491274    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.491370 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491308    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.492405 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.491982    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.492910 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.492620    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.494806 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.494778    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.495142 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.495110    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.496677 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.495651    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.496677 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.496373    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.496677 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.496444    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9898e745-053c-4cdf-89b8-a0a93dad3b8d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.496995 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.496975    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9898e745-053c-4cdf-89b8-a0a93dad3b8d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.498327 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.498004    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.498327 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.498223    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.498327 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.498268    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.498327 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.498290    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9898e745-053c-4cdf-89b8-a0a93dad3b8d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.498757 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.498737    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.498849 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.498829    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.499284 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.499264    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.500235 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.500195    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9898e745-053c-4cdf-89b8-a0a93dad3b8d-config\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.500314 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.500235    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9898e745-053c-4cdf-89b8-a0a93dad3b8d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.502358 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.502336    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jld5c\" (UniqueName: \"kubernetes.io/projected/9898e745-053c-4cdf-89b8-a0a93dad3b8d-kube-api-access-jld5c\") pod \"prometheus-k8s-0\" (UID: \"9898e745-053c-4cdf-89b8-a0a93dad3b8d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.547984 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.547949    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:49:15.708972 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.708921    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 02 15:49:15.711291 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:49:15.711259    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9898e745_053c_4cdf_89b8_a0a93dad3b8d.slice/crio-c73e570913d1a434e3a3d60d7a52c8ed1b49c9629fdc297af63da40717e69063 WatchSource:0}: Error finding container c73e570913d1a434e3a3d60d7a52c8ed1b49c9629fdc297af63da40717e69063: Status 404 returned error can't find the container with id c73e570913d1a434e3a3d60d7a52c8ed1b49c9629fdc297af63da40717e69063
Jun 02 15:49:15.872627 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.872584    2577 generic.go:358] "Generic (PLEG): container finished" podID="9898e745-053c-4cdf-89b8-a0a93dad3b8d" containerID="d4d8a3e02478e7ecaf22f1c3fcc38c061f038bb2b62df7831b8da206ec98de45" exitCode=0
Jun 02 15:49:15.873106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.872634    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9898e745-053c-4cdf-89b8-a0a93dad3b8d","Type":"ContainerDied","Data":"d4d8a3e02478e7ecaf22f1c3fcc38c061f038bb2b62df7831b8da206ec98de45"}
Jun 02 15:49:15.873106 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:15.872691    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9898e745-053c-4cdf-89b8-a0a93dad3b8d","Type":"ContainerStarted","Data":"c73e570913d1a434e3a3d60d7a52c8ed1b49c9629fdc297af63da40717e69063"}
Jun 02 15:49:16.226281 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.226245    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb5f110-9389-4545-b0a7-b6f0a7d79aa0" path="/var/lib/kubelet/pods/8cb5f110-9389-4545-b0a7-b6f0a7d79aa0/volumes"
Jun 02 15:49:16.878038 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.877998    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp" event={"ID":"0e1a51e0-f52e-4d54-8d67-dad858c300de","Type":"ContainerStarted","Data":"ea572e1fb56bf37335fdf71e170b22ffc7e1b89eafea68bca3864d4cad0fc900"}
Jun 02 15:49:16.878038 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.878043    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp" event={"ID":"0e1a51e0-f52e-4d54-8d67-dad858c300de","Type":"ContainerStarted","Data":"9787c7c1eb9f0dc9710861cecd58cc488b552ffd2f2aac6d4d0e6f64ca0ef52a"}
Jun 02 15:49:16.878558 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.878058    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp" event={"ID":"0e1a51e0-f52e-4d54-8d67-dad858c300de","Type":"ContainerStarted","Data":"9fb7613d10138ae96d7c508cb99d736766b530cf04ff54f3f5afcf50360a2bde"}
Jun 02 15:49:16.880736 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.880713    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9898e745-053c-4cdf-89b8-a0a93dad3b8d","Type":"ContainerStarted","Data":"0dcd25bc9fd2d9fa398186f954439b934ea25c70d10ef41ad55c53fa0acb3dc1"}
Jun 02 15:49:16.880736 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.880740    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9898e745-053c-4cdf-89b8-a0a93dad3b8d","Type":"ContainerStarted","Data":"a0633c27b1dc2bf0120e3719bb1102a6c4d8410327455a0959365102f275a4f1"}
Jun 02 15:49:16.880880 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.880750    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9898e745-053c-4cdf-89b8-a0a93dad3b8d","Type":"ContainerStarted","Data":"eb5200edb1d1403ab78d3c952f916e1a99036890d00fb6a284a191d2ba47c3b0"}
Jun 02 15:49:16.880880 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.880758    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9898e745-053c-4cdf-89b8-a0a93dad3b8d","Type":"ContainerStarted","Data":"35e8fdedf0da4605ce01a78d5b3dbf6930e27330354e89b918b6af04967ef36d"}
Jun 02 15:49:16.880880 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.880767    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9898e745-053c-4cdf-89b8-a0a93dad3b8d","Type":"ContainerStarted","Data":"7a808ed27c5bd0411dceaf48ca959339fa04b1905241e2bca537a884aca9dd25"}
Jun 02 15:49:16.880880 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.880775    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9898e745-053c-4cdf-89b8-a0a93dad3b8d","Type":"ContainerStarted","Data":"ad8752404fcc61c85c55a951fea018e09b560d8a6c89b00084998359027e809f"}
Jun 02 15:49:16.903775 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.903716    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6dc5d86cb8-b8vrp" podStartSLOduration=1.143327605 podStartE2EDuration="2.903701087s" podCreationTimestamp="2026-06-02 15:49:14 +0000 UTC" firstStartedPulling="2026-06-02 15:49:14.673618428 +0000 UTC m=+159.016039627" lastFinishedPulling="2026-06-02 15:49:16.433991893 +0000 UTC m=+160.776413109" observedRunningTime="2026-06-02 15:49:16.901547592 +0000 UTC m=+161.243968814" watchObservedRunningTime="2026-06-02 15:49:16.903701087 +0000 UTC m=+161.246122310"
Jun 02 15:49:16.930852 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:16.930809    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.930794119 podStartE2EDuration="1.930794119s" podCreationTimestamp="2026-06-02 15:49:15 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:49:16.928583933 +0000 UTC m=+161.271005155" watchObservedRunningTime="2026-06-02 15:49:16.930794119 +0000 UTC m=+161.273215341"
Jun 02 15:49:20.548666 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:49:20.548617    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:50:15.548649 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:50:15.548608    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:50:15.564373 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:50:15.564344    2577 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:50:16.068050 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:50:16.068021    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 02 15:51:09.787624 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.787535    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x5r6b"]
Jun 02 15:51:09.790569 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.790548    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.792638 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.792620    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 02 15:51:09.800877 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.800850    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5r6b"]
Jun 02 15:51:09.849064 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.849034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eae3712a-0e98-45de-a57a-3f0f8ae67289-dbus\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.849249 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.849079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eae3712a-0e98-45de-a57a-3f0f8ae67289-kubelet-config\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.849249 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.849109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eae3712a-0e98-45de-a57a-3f0f8ae67289-original-pull-secret\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.949968 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.949917    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eae3712a-0e98-45de-a57a-3f0f8ae67289-dbus\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.950140 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.949984    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eae3712a-0e98-45de-a57a-3f0f8ae67289-kubelet-config\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.950140 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.950013    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eae3712a-0e98-45de-a57a-3f0f8ae67289-original-pull-secret\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.950140 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.950102    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eae3712a-0e98-45de-a57a-3f0f8ae67289-dbus\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.950140 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.950120    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eae3712a-0e98-45de-a57a-3f0f8ae67289-kubelet-config\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:09.952315 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:09.952288    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eae3712a-0e98-45de-a57a-3f0f8ae67289-original-pull-secret\") pod \"global-pull-secret-syncer-x5r6b\" (UID: \"eae3712a-0e98-45de-a57a-3f0f8ae67289\") " pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:10.099284 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:10.099193    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x5r6b"
Jun 02 15:51:10.224242 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:10.224188    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x5r6b"]
Jun 02 15:51:10.227700 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:51:10.227670    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeae3712a_0e98_45de_a57a_3f0f8ae67289.slice/crio-63fb3bd5964171c8ec0e590d688efcc3862ec7571131dd71ba14e61cb4fde1bc WatchSource:0}: Error finding container 63fb3bd5964171c8ec0e590d688efcc3862ec7571131dd71ba14e61cb4fde1bc: Status 404 returned error can't find the container with id 63fb3bd5964171c8ec0e590d688efcc3862ec7571131dd71ba14e61cb4fde1bc
Jun 02 15:51:11.223142 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:11.223105    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5r6b" event={"ID":"eae3712a-0e98-45de-a57a-3f0f8ae67289","Type":"ContainerStarted","Data":"63fb3bd5964171c8ec0e590d688efcc3862ec7571131dd71ba14e61cb4fde1bc"}
Jun 02 15:51:15.239005 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:15.238969    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x5r6b" event={"ID":"eae3712a-0e98-45de-a57a-3f0f8ae67289","Type":"ContainerStarted","Data":"f013e055e16c02769ba99cbd7913e340a1ff2d1d274d09182004827a6ceb485a"}
Jun 02 15:51:15.257454 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:15.257405    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x5r6b" podStartSLOduration=2.294625147 podStartE2EDuration="6.257373361s" podCreationTimestamp="2026-06-02 15:51:09 +0000 UTC" firstStartedPulling="2026-06-02 15:51:10.229555865 +0000 UTC m=+274.571977068" lastFinishedPulling="2026-06-02 15:51:14.192304081 +0000 UTC m=+278.534725282" observedRunningTime="2026-06-02 15:51:15.256027701 +0000 UTC m=+279.598448923" watchObservedRunningTime="2026-06-02 15:51:15.257373361 +0000 UTC m=+279.599794583"
Jun 02 15:51:36.109481 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:51:36.109445    2577 kubelet.go:1628] "Image garbage collection succeeded"
Jun 02 15:53:50.608173 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.608138    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-79f457656b-bp58r"]
Jun 02 15:53:50.610904 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.610889    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:50.614107 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.614085    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 02 15:53:50.614697 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.614681    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 02 15:53:50.614791 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.614683    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jun 02 15:53:50.614791 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.614684    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wv8ft\""
Jun 02 15:53:50.625403 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.625373    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-bp58r"]
Jun 02 15:53:50.701295 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.701267    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qw4l\" (UniqueName: \"kubernetes.io/projected/fb67966a-cb68-40d4-8b42-f97c928c540e-kube-api-access-9qw4l\") pod \"seaweedfs-79f457656b-bp58r\" (UID: \"fb67966a-cb68-40d4-8b42-f97c928c540e\") " pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:50.701478 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.701318    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fb67966a-cb68-40d4-8b42-f97c928c540e-data\") pod \"seaweedfs-79f457656b-bp58r\" (UID: \"fb67966a-cb68-40d4-8b42-f97c928c540e\") " pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:50.802207 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.802170    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9qw4l\" (UniqueName: \"kubernetes.io/projected/fb67966a-cb68-40d4-8b42-f97c928c540e-kube-api-access-9qw4l\") pod \"seaweedfs-79f457656b-bp58r\" (UID: \"fb67966a-cb68-40d4-8b42-f97c928c540e\") " pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:50.802422 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.802235    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fb67966a-cb68-40d4-8b42-f97c928c540e-data\") pod \"seaweedfs-79f457656b-bp58r\" (UID: \"fb67966a-cb68-40d4-8b42-f97c928c540e\") " pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:50.802605 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.802590    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fb67966a-cb68-40d4-8b42-f97c928c540e-data\") pod \"seaweedfs-79f457656b-bp58r\" (UID: \"fb67966a-cb68-40d4-8b42-f97c928c540e\") " pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:50.812680 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.812657    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qw4l\" (UniqueName: \"kubernetes.io/projected/fb67966a-cb68-40d4-8b42-f97c928c540e-kube-api-access-9qw4l\") pod \"seaweedfs-79f457656b-bp58r\" (UID: \"fb67966a-cb68-40d4-8b42-f97c928c540e\") " pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:50.920092 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:50.920063    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:51.047959 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:51.047923    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-79f457656b-bp58r"]
Jun 02 15:53:51.051009 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:53:51.050984    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb67966a_cb68_40d4_8b42_f97c928c540e.slice/crio-f0dde2fb2959154f63f23c67e86ea41eed7ad830abfabbbf1a662afdba6a6744 WatchSource:0}: Error finding container f0dde2fb2959154f63f23c67e86ea41eed7ad830abfabbbf1a662afdba6a6744: Status 404 returned error can't find the container with id f0dde2fb2959154f63f23c67e86ea41eed7ad830abfabbbf1a662afdba6a6744
Jun 02 15:53:51.052642 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:51.052627    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:53:51.681496 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:51.681441    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-bp58r" event={"ID":"fb67966a-cb68-40d4-8b42-f97c928c540e","Type":"ContainerStarted","Data":"f0dde2fb2959154f63f23c67e86ea41eed7ad830abfabbbf1a662afdba6a6744"}
Jun 02 15:53:55.694977 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:55.694943    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-79f457656b-bp58r" event={"ID":"fb67966a-cb68-40d4-8b42-f97c928c540e","Type":"ContainerStarted","Data":"32b6990004f6770ace0e8ee31d77deb998e9f9907de08158ee7880166ba3567d"}
Jun 02 15:53:55.695363 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:55.694994    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:53:55.711721 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:53:55.711672    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/seaweedfs-79f457656b-bp58r" podStartSLOduration=1.432081073 podStartE2EDuration="5.711654799s" podCreationTimestamp="2026-06-02 15:53:50 +0000 UTC" firstStartedPulling="2026-06-02 15:53:51.052747298 +0000 UTC m=+435.395168498" lastFinishedPulling="2026-06-02 15:53:55.33232102 +0000 UTC m=+439.674742224" observedRunningTime="2026-06-02 15:53:55.710853215 +0000 UTC m=+440.053274437" watchObservedRunningTime="2026-06-02 15:53:55.711654799 +0000 UTC m=+440.054076023"
Jun 02 15:54:01.699867 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:54:01.699837    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-79f457656b-bp58r"
Jun 02 15:55:16.127109 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.127060    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-2dxcd"]
Jun 02 15:55:16.130852 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.130832    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2dxcd"
Jun 02 15:55:16.136338 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.136308    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2dxcd"]
Jun 02 15:55:16.150837 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.150805    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jcb\" (UniqueName: \"kubernetes.io/projected/61707792-f927-4b6e-9216-00fe50a3d306-kube-api-access-v2jcb\") pod \"s3-init-2dxcd\" (UID: \"61707792-f927-4b6e-9216-00fe50a3d306\") " pod="kserve/s3-init-2dxcd"
Jun 02 15:55:16.251318 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.251289    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jcb\" (UniqueName: \"kubernetes.io/projected/61707792-f927-4b6e-9216-00fe50a3d306-kube-api-access-v2jcb\") pod \"s3-init-2dxcd\" (UID: \"61707792-f927-4b6e-9216-00fe50a3d306\") " pod="kserve/s3-init-2dxcd"
Jun 02 15:55:16.262288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.262255    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jcb\" (UniqueName: \"kubernetes.io/projected/61707792-f927-4b6e-9216-00fe50a3d306-kube-api-access-v2jcb\") pod \"s3-init-2dxcd\" (UID: \"61707792-f927-4b6e-9216-00fe50a3d306\") " pod="kserve/s3-init-2dxcd"
Jun 02 15:55:16.455979 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.455941    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2dxcd"
Jun 02 15:55:16.589797 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.589645    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-2dxcd"]
Jun 02 15:55:16.592197 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:55:16.592165    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61707792_f927_4b6e_9216_00fe50a3d306.slice/crio-4a884704d4a2554cf155981f4f4d6a29237f9433b52c0eddaec930fc19a503bd WatchSource:0}: Error finding container 4a884704d4a2554cf155981f4f4d6a29237f9433b52c0eddaec930fc19a503bd: Status 404 returned error can't find the container with id 4a884704d4a2554cf155981f4f4d6a29237f9433b52c0eddaec930fc19a503bd
Jun 02 15:55:16.935271 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:16.935235    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2dxcd" event={"ID":"61707792-f927-4b6e-9216-00fe50a3d306","Type":"ContainerStarted","Data":"4a884704d4a2554cf155981f4f4d6a29237f9433b52c0eddaec930fc19a503bd"}
Jun 02 15:55:21.952862 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:21.952816    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2dxcd" event={"ID":"61707792-f927-4b6e-9216-00fe50a3d306","Type":"ContainerStarted","Data":"95abebc54c4c91f611f0145b44b9deb4fe82a7867b9648a598ac91df1c2f4f0a"}
Jun 02 15:55:21.969421 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:21.969334    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-2dxcd" podStartSLOduration=1.5035387180000002 podStartE2EDuration="5.969314825s" podCreationTimestamp="2026-06-02 15:55:16 +0000 UTC" firstStartedPulling="2026-06-02 15:55:16.594284156 +0000 UTC m=+520.936705360" lastFinishedPulling="2026-06-02 15:55:21.060060265 +0000 UTC m=+525.402481467" observedRunningTime="2026-06-02 15:55:21.969115571 +0000 UTC m=+526.311536792" watchObservedRunningTime="2026-06-02 15:55:21.969314825 +0000 UTC m=+526.311736047"
Jun 02 15:55:24.962372 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:24.962338    2577 generic.go:358] "Generic (PLEG): container finished" podID="61707792-f927-4b6e-9216-00fe50a3d306" containerID="95abebc54c4c91f611f0145b44b9deb4fe82a7867b9648a598ac91df1c2f4f0a" exitCode=0
Jun 02 15:55:24.962787 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:24.962411    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2dxcd" event={"ID":"61707792-f927-4b6e-9216-00fe50a3d306","Type":"ContainerDied","Data":"95abebc54c4c91f611f0145b44b9deb4fe82a7867b9648a598ac91df1c2f4f0a"}
Jun 02 15:55:26.090381 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:26.090359    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2dxcd"
Jun 02 15:55:26.147477 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:26.147440    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2jcb\" (UniqueName: \"kubernetes.io/projected/61707792-f927-4b6e-9216-00fe50a3d306-kube-api-access-v2jcb\") pod \"61707792-f927-4b6e-9216-00fe50a3d306\" (UID: \"61707792-f927-4b6e-9216-00fe50a3d306\") "
Jun 02 15:55:26.149544 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:26.149513    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61707792-f927-4b6e-9216-00fe50a3d306-kube-api-access-v2jcb" (OuterVolumeSpecName: "kube-api-access-v2jcb") pod "61707792-f927-4b6e-9216-00fe50a3d306" (UID: "61707792-f927-4b6e-9216-00fe50a3d306"). InnerVolumeSpecName "kube-api-access-v2jcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:55:26.248001 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:26.247967    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2jcb\" (UniqueName: \"kubernetes.io/projected/61707792-f927-4b6e-9216-00fe50a3d306-kube-api-access-v2jcb\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:55:26.969294 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:26.969267    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-2dxcd"
Jun 02 15:55:26.969294 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:26.969279    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-2dxcd" event={"ID":"61707792-f927-4b6e-9216-00fe50a3d306","Type":"ContainerDied","Data":"4a884704d4a2554cf155981f4f4d6a29237f9433b52c0eddaec930fc19a503bd"}
Jun 02 15:55:26.969523 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:26.969309    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a884704d4a2554cf155981f4f4d6a29237f9433b52c0eddaec930fc19a503bd"
Jun 02 15:55:36.454669 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.454633    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"]
Jun 02 15:55:36.455154 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.454973    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61707792-f927-4b6e-9216-00fe50a3d306" containerName="s3-init"
Jun 02 15:55:36.455154 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.454986    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="61707792-f927-4b6e-9216-00fe50a3d306" containerName="s3-init"
Jun 02 15:55:36.455154 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.455051    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="61707792-f927-4b6e-9216-00fe50a3d306" containerName="s3-init"
Jun 02 15:55:36.458452 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.458379    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.461024 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.460996    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jun 02 15:55:36.461148 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.461124    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g84l2\""
Jun 02 15:55:36.462061 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.462041    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 02 15:55:36.462183 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.462134    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jun 02 15:55:36.462657 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.462631    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 02 15:55:36.469086 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.469069    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"]
Jun 02 15:55:36.536984 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.536953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.537167 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.537027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.537167 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.537056    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.537167 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.537080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxmwj\" (UniqueName: \"kubernetes.io/projected/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kube-api-access-pxmwj\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.542592 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.542568    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"]
Jun 02 15:55:36.545703 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.545688    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.547910 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.547891    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-baff1-predictor-serving-cert\""
Jun 02 15:55:36.548006 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.547892    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-baff1-kube-rbac-proxy-sar-config\""
Jun 02 15:55:36.556021 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.555999    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"]
Jun 02 15:55:36.638048 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638011    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-baff1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb12859-245c-4653-a61b-80a45dc8a55c-error-404-isvc-baff1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.638215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638057    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb12859-245c-4653-a61b-80a45dc8a55c-proxy-tls\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.638215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638140    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.638215 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638184    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5cg\" (UniqueName: \"kubernetes.io/projected/9bb12859-245c-4653-a61b-80a45dc8a55c-kube-api-access-nd5cg\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.638320 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638225    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.638374 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638351    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.638451 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638429    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pxmwj\" (UniqueName: \"kubernetes.io/projected/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kube-api-access-pxmwj\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.638747 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638727    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.638912 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.638896    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.640674 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.640651    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.647622 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.647602    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxmwj\" (UniqueName: \"kubernetes.io/projected/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kube-api-access-pxmwj\") pod \"isvc-sklearn-graph-1-predictor-55cb655557-fr524\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.667761 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.667738    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"]
Jun 02 15:55:36.671457 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.671439    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.673650 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.673631    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jun 02 15:55:36.673804 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.673786    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jun 02 15:55:36.682321 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.682299    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"]
Jun 02 15:55:36.739332 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.739236    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46kq7\" (UniqueName: \"kubernetes.io/projected/36beb25a-880a-4154-9112-9be2c2456cc1-kube-api-access-46kq7\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.739332 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.739281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36beb25a-880a-4154-9112-9be2c2456cc1-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.739578 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.739378    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-baff1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb12859-245c-4653-a61b-80a45dc8a55c-error-404-isvc-baff1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.739578 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.739438    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb12859-245c-4653-a61b-80a45dc8a55c-proxy-tls\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.739578 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.739461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36beb25a-880a-4154-9112-9be2c2456cc1-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.739578 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.739500    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5cg\" (UniqueName: \"kubernetes.io/projected/9bb12859-245c-4653-a61b-80a45dc8a55c-kube-api-access-nd5cg\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.739578 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.739530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36beb25a-880a-4154-9112-9be2c2456cc1-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.740100 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.740078    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-baff1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb12859-245c-4653-a61b-80a45dc8a55c-error-404-isvc-baff1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.741863 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.741846    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb12859-245c-4653-a61b-80a45dc8a55c-proxy-tls\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.748378 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.748354    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5cg\" (UniqueName: \"kubernetes.io/projected/9bb12859-245c-4653-a61b-80a45dc8a55c-kube-api-access-nd5cg\") pod \"error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") " pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.769720 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.769696    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:55:36.840627 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.840595    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36beb25a-880a-4154-9112-9be2c2456cc1-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.840806 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.840660    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-46kq7\" (UniqueName: \"kubernetes.io/projected/36beb25a-880a-4154-9112-9be2c2456cc1-kube-api-access-46kq7\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.840806 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.840694    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36beb25a-880a-4154-9112-9be2c2456cc1-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.840806 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.840764    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36beb25a-880a-4154-9112-9be2c2456cc1-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.841225 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.841200    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36beb25a-880a-4154-9112-9be2c2456cc1-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.841631 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.841613    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36beb25a-880a-4154-9112-9be2c2456cc1-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.843198 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.843180    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36beb25a-880a-4154-9112-9be2c2456cc1-proxy-tls\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.850011 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.849983    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46kq7\" (UniqueName: \"kubernetes.io/projected/36beb25a-880a-4154-9112-9be2c2456cc1-kube-api-access-46kq7\") pod \"isvc-xgboost-graph-predictor-669d8d6456-svbpc\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.855871 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.855844    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:36.910297 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.910248    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"]
Jun 02 15:55:36.913786 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:55:36.913648    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ad842b_0ad1_463a_86ee_a4e55a5e9eba.slice/crio-53afeaf2593d8a2f17a60e6adc31bddb9d14abd946ac6e85fa09ff9a1939ebc5 WatchSource:0}: Error finding container 53afeaf2593d8a2f17a60e6adc31bddb9d14abd946ac6e85fa09ff9a1939ebc5: Status 404 returned error can't find the container with id 53afeaf2593d8a2f17a60e6adc31bddb9d14abd946ac6e85fa09ff9a1939ebc5
Jun 02 15:55:36.981881 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.981850    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:55:36.998461 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.998426    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" event={"ID":"42ad842b-0ad1-463a-86ee-a4e55a5e9eba","Type":"ContainerStarted","Data":"53afeaf2593d8a2f17a60e6adc31bddb9d14abd946ac6e85fa09ff9a1939ebc5"}
Jun 02 15:55:36.998882 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:36.998864    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"]
Jun 02 15:55:37.001783 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:55:37.001750    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bb12859_245c_4653_a61b_80a45dc8a55c.slice/crio-0e52b4f5d3d0ee528a3db6e7ecf9e10f78578b8a2e72d00b949eb0bcaf230b79 WatchSource:0}: Error finding container 0e52b4f5d3d0ee528a3db6e7ecf9e10f78578b8a2e72d00b949eb0bcaf230b79: Status 404 returned error can't find the container with id 0e52b4f5d3d0ee528a3db6e7ecf9e10f78578b8a2e72d00b949eb0bcaf230b79
Jun 02 15:55:37.111842 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:37.111810    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"]
Jun 02 15:55:37.116438 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:55:37.115504    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36beb25a_880a_4154_9112_9be2c2456cc1.slice/crio-c1502b87f1156ecf5dcde398a126dd18dc4d7bfdbc7c4e34d55b5b27863c8edb WatchSource:0}: Error finding container c1502b87f1156ecf5dcde398a126dd18dc4d7bfdbc7c4e34d55b5b27863c8edb: Status 404 returned error can't find the container with id c1502b87f1156ecf5dcde398a126dd18dc4d7bfdbc7c4e34d55b5b27863c8edb
Jun 02 15:55:38.006556 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:38.006509    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" event={"ID":"9bb12859-245c-4653-a61b-80a45dc8a55c","Type":"ContainerStarted","Data":"0e52b4f5d3d0ee528a3db6e7ecf9e10f78578b8a2e72d00b949eb0bcaf230b79"}
Jun 02 15:55:38.012885 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:38.012807    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" event={"ID":"36beb25a-880a-4154-9112-9be2c2456cc1","Type":"ContainerStarted","Data":"c1502b87f1156ecf5dcde398a126dd18dc4d7bfdbc7c4e34d55b5b27863c8edb"}
Jun 02 15:55:51.077643 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:51.077603    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" event={"ID":"36beb25a-880a-4154-9112-9be2c2456cc1","Type":"ContainerStarted","Data":"070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3"}
Jun 02 15:55:51.079170 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:51.079129    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" event={"ID":"42ad842b-0ad1-463a-86ee-a4e55a5e9eba","Type":"ContainerStarted","Data":"3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3"}
Jun 02 15:55:51.080458 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:51.080435    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" event={"ID":"9bb12859-245c-4653-a61b-80a45dc8a55c","Type":"ContainerStarted","Data":"6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3"}
Jun 02 15:55:54.092829 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:54.092782    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" event={"ID":"9bb12859-245c-4653-a61b-80a45dc8a55c","Type":"ContainerStarted","Data":"ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1"}
Jun 02 15:55:54.093242 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:54.092946    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:54.112948 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:54.112892    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podStartSLOduration=1.6107885899999999 podStartE2EDuration="18.112876617s" podCreationTimestamp="2026-06-02 15:55:36 +0000 UTC" firstStartedPulling="2026-06-02 15:55:37.003783655 +0000 UTC m=+541.346204855" lastFinishedPulling="2026-06-02 15:55:53.505871681 +0000 UTC m=+557.848292882" observedRunningTime="2026-06-02 15:55:54.111845086 +0000 UTC m=+558.454266308" watchObservedRunningTime="2026-06-02 15:55:54.112876617 +0000 UTC m=+558.455297839"
Jun 02 15:55:55.099780 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:55.099747    2577 generic.go:358] "Generic (PLEG): container finished" podID="36beb25a-880a-4154-9112-9be2c2456cc1" containerID="070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3" exitCode=0
Jun 02 15:55:55.100185 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:55.099820    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" event={"ID":"36beb25a-880a-4154-9112-9be2c2456cc1","Type":"ContainerDied","Data":"070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3"}
Jun 02 15:55:55.101264 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:55.101243    2577 generic.go:358] "Generic (PLEG): container finished" podID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerID="3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3" exitCode=0
Jun 02 15:55:55.101348 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:55.101298    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" event={"ID":"42ad842b-0ad1-463a-86ee-a4e55a5e9eba","Type":"ContainerDied","Data":"3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3"}
Jun 02 15:55:55.101564 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:55.101535    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:55:55.102955 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:55.102927    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 02 15:55:56.106013 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:55:56.105956    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 02 15:56:01.111784 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:01.111750    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:56:01.113018 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:01.112634    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 02 15:56:03.133663 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:03.133594    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" event={"ID":"42ad842b-0ad1-463a-86ee-a4e55a5e9eba","Type":"ContainerStarted","Data":"eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec"}
Jun 02 15:56:03.133663 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:03.133666    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" event={"ID":"42ad842b-0ad1-463a-86ee-a4e55a5e9eba","Type":"ContainerStarted","Data":"0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2"}
Jun 02 15:56:03.134476 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:03.134451    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:56:03.134623 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:03.134486    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:56:03.135266 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:03.135239    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:56:03.156490 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:03.156430    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podStartSLOduration=1.9004270760000002 podStartE2EDuration="27.156411106s" podCreationTimestamp="2026-06-02 15:55:36 +0000 UTC" firstStartedPulling="2026-06-02 15:55:36.915753509 +0000 UTC m=+541.258174723" lastFinishedPulling="2026-06-02 15:56:02.171737554 +0000 UTC m=+566.514158753" observedRunningTime="2026-06-02 15:56:03.153638224 +0000 UTC m=+567.496059469" watchObservedRunningTime="2026-06-02 15:56:03.156411106 +0000 UTC m=+567.498832331"
Jun 02 15:56:04.137057 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:04.137017    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:56:09.142182 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:09.142151    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:56:09.142866 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:09.142827    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:56:11.113519 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:11.113476    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 02 15:56:15.189474 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:15.189429    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" event={"ID":"36beb25a-880a-4154-9112-9be2c2456cc1","Type":"ContainerStarted","Data":"4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907"}
Jun 02 15:56:15.189943 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:15.189486    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" event={"ID":"36beb25a-880a-4154-9112-9be2c2456cc1","Type":"ContainerStarted","Data":"0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374"}
Jun 02 15:56:15.189943 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:15.189887    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:56:15.190054 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:15.189997    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:56:15.191244 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:15.191212    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:56:15.209446 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:15.209377    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podStartSLOduration=2.117372327 podStartE2EDuration="39.209365363s" podCreationTimestamp="2026-06-02 15:55:36 +0000 UTC" firstStartedPulling="2026-06-02 15:55:37.118552001 +0000 UTC m=+541.460973201" lastFinishedPulling="2026-06-02 15:56:14.210545037 +0000 UTC m=+578.552966237" observedRunningTime="2026-06-02 15:56:15.208612137 +0000 UTC m=+579.551033360" watchObservedRunningTime="2026-06-02 15:56:15.209365363 +0000 UTC m=+579.551786585"
Jun 02 15:56:16.192708 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:16.192662    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:56:19.143044 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:19.143006    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:56:21.113320 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:21.113278    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 02 15:56:21.197076 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:21.197047    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:56:21.197698 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:21.197673    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:56:29.143353 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:29.143311    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:56:31.113404 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:31.113362    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 02 15:56:31.198257 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:31.198214    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:56:39.142701 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:39.142662    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:56:41.113305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:41.113269    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:56:41.197607 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:41.197567    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:56:49.143628 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:49.143583    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:56:51.198208 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:51.198163    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:56:59.143288 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:56:59.143193    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:57:01.197614 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:01.197577    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:57:09.143327 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:09.143288    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:57:10.667496 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.667441    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"]
Jun 02 15:57:10.667861 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.667748    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container" containerID="cri-o://6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3" gracePeriod=30
Jun 02 15:57:10.667861 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.667802    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kube-rbac-proxy" containerID="cri-o://ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1" gracePeriod=30
Jun 02 15:57:10.684744 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.684720    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"]
Jun 02 15:57:10.687148 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.687129    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.689299 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.689275    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c6170-predictor-serving-cert\""
Jun 02 15:57:10.689417 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.689281    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c6170-kube-rbac-proxy-sar-config\""
Jun 02 15:57:10.700008 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.699987    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"]
Jun 02 15:57:10.820717 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.820669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-c6170-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-error-404-isvc-c6170-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.820717 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.820719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-proxy-tls\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.820936 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.820738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvdm\" (UniqueName: \"kubernetes.io/projected/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-kube-api-access-gmvdm\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.921518 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.921420    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-c6170-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-error-404-isvc-c6170-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.921518 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.921460    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-proxy-tls\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.921518 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.921498    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvdm\" (UniqueName: \"kubernetes.io/projected/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-kube-api-access-gmvdm\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.922130 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.922106    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-c6170-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-error-404-isvc-c6170-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.923869 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.923841    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-proxy-tls\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.933337 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.933313    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvdm\" (UniqueName: \"kubernetes.io/projected/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-kube-api-access-gmvdm\") pod \"error-404-isvc-c6170-predictor-59b777545b-kglp9\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") " pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:10.997640 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:10.997610    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:11.107047 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.107004    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jun 02 15:57:11.113606 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.113569    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jun 02 15:57:11.129211 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.129154    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"]
Jun 02 15:57:11.134192 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:57:11.134160    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c39ca9_cd23_4430_9d5c_f0fb1a4c3841.slice/crio-a777025fec0646e8b6066a61d016d63f550d17ccb36131e2b9e926f435516176 WatchSource:0}: Error finding container a777025fec0646e8b6066a61d016d63f550d17ccb36131e2b9e926f435516176: Status 404 returned error can't find the container with id a777025fec0646e8b6066a61d016d63f550d17ccb36131e2b9e926f435516176
Jun 02 15:57:11.197742 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.197711    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jun 02 15:57:11.358735 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.358699    2577 generic.go:358] "Generic (PLEG): container finished" podID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerID="ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1" exitCode=2
Jun 02 15:57:11.358919 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.358758    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" event={"ID":"9bb12859-245c-4653-a61b-80a45dc8a55c","Type":"ContainerDied","Data":"ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1"}
Jun 02 15:57:11.360187 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.360159    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" event={"ID":"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841","Type":"ContainerStarted","Data":"3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7"}
Jun 02 15:57:11.360305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.360193    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" event={"ID":"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841","Type":"ContainerStarted","Data":"1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112"}
Jun 02 15:57:11.360305 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.360206    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" event={"ID":"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841","Type":"ContainerStarted","Data":"a777025fec0646e8b6066a61d016d63f550d17ccb36131e2b9e926f435516176"}
Jun 02 15:57:11.360425 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.360408    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:11.381232 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:11.381184    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podStartSLOduration=1.381165985 podStartE2EDuration="1.381165985s" podCreationTimestamp="2026-06-02 15:57:10 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:57:11.378797712 +0000 UTC m=+635.721218934" watchObservedRunningTime="2026-06-02 15:57:11.381165985 +0000 UTC m=+635.723587229"
Jun 02 15:57:12.364126 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:12.364095    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:12.365357 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:12.365330    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 15:57:13.367674 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:13.367587    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 15:57:14.003713 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.003689    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:57:14.150204 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.150176    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd5cg\" (UniqueName: \"kubernetes.io/projected/9bb12859-245c-4653-a61b-80a45dc8a55c-kube-api-access-nd5cg\") pod \"9bb12859-245c-4653-a61b-80a45dc8a55c\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") "
Jun 02 15:57:14.150406 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.150234    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb12859-245c-4653-a61b-80a45dc8a55c-proxy-tls\") pod \"9bb12859-245c-4653-a61b-80a45dc8a55c\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") "
Jun 02 15:57:14.150406 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.150304    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-baff1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb12859-245c-4653-a61b-80a45dc8a55c-error-404-isvc-baff1-kube-rbac-proxy-sar-config\") pod \"9bb12859-245c-4653-a61b-80a45dc8a55c\" (UID: \"9bb12859-245c-4653-a61b-80a45dc8a55c\") "
Jun 02 15:57:14.150698 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.150666    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bb12859-245c-4653-a61b-80a45dc8a55c-error-404-isvc-baff1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-baff1-kube-rbac-proxy-sar-config") pod "9bb12859-245c-4653-a61b-80a45dc8a55c" (UID: "9bb12859-245c-4653-a61b-80a45dc8a55c"). InnerVolumeSpecName "error-404-isvc-baff1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:57:14.152500 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.152475    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb12859-245c-4653-a61b-80a45dc8a55c-kube-api-access-nd5cg" (OuterVolumeSpecName: "kube-api-access-nd5cg") pod "9bb12859-245c-4653-a61b-80a45dc8a55c" (UID: "9bb12859-245c-4653-a61b-80a45dc8a55c"). InnerVolumeSpecName "kube-api-access-nd5cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:57:14.152579 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.152523    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb12859-245c-4653-a61b-80a45dc8a55c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9bb12859-245c-4653-a61b-80a45dc8a55c" (UID: "9bb12859-245c-4653-a61b-80a45dc8a55c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:57:14.251882 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.251852    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9bb12859-245c-4653-a61b-80a45dc8a55c-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:57:14.251882 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.251878    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-baff1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9bb12859-245c-4653-a61b-80a45dc8a55c-error-404-isvc-baff1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:57:14.252051 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.251889    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd5cg\" (UniqueName: \"kubernetes.io/projected/9bb12859-245c-4653-a61b-80a45dc8a55c-kube-api-access-nd5cg\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:57:14.371191 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.371161    2577 generic.go:358] "Generic (PLEG): container finished" podID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerID="6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3" exitCode=0
Jun 02 15:57:14.371699 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.371218    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" event={"ID":"9bb12859-245c-4653-a61b-80a45dc8a55c","Type":"ContainerDied","Data":"6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3"}
Jun 02 15:57:14.371699 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.371239    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"
Jun 02 15:57:14.371699 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.371255    2577 scope.go:117] "RemoveContainer" containerID="ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1"
Jun 02 15:57:14.371699 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.371244    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j" event={"ID":"9bb12859-245c-4653-a61b-80a45dc8a55c","Type":"ContainerDied","Data":"0e52b4f5d3d0ee528a3db6e7ecf9e10f78578b8a2e72d00b949eb0bcaf230b79"}
Jun 02 15:57:14.379198 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.379179    2577 scope.go:117] "RemoveContainer" containerID="6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3"
Jun 02 15:57:14.386556 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.386425    2577 scope.go:117] "RemoveContainer" containerID="ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1"
Jun 02 15:57:14.386729 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:57:14.386709    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1\": container with ID starting with ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1 not found: ID does not exist" containerID="ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1"
Jun 02 15:57:14.386800 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.386736    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1"} err="failed to get container status \"ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1\": rpc error: code = NotFound desc = could not find container \"ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1\": container with ID starting with ff9215a09d27cf2f606db27fffc28d50855ea958dccb75f995c2c51381019bb1 not found: ID does not exist"
Jun 02 15:57:14.386800 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.386755    2577 scope.go:117] "RemoveContainer" containerID="6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3"
Jun 02 15:57:14.387001 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:57:14.386977    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3\": container with ID starting with 6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3 not found: ID does not exist" containerID="6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3"
Jun 02 15:57:14.387061 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.387008    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3"} err="failed to get container status \"6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3\": rpc error: code = NotFound desc = could not find container \"6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3\": container with ID starting with 6d20844eed9da449c49bc622ec3e659f5d22a952b3509b8cb298049d95ddbcb3 not found: ID does not exist"
Jun 02 15:57:14.394507 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.394482    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"]
Jun 02 15:57:14.400274 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:14.400216    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-baff1-predictor-6c8f9d8cc4-pwd6j"]
Jun 02 15:57:16.222841 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:16.222806    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" path="/var/lib/kubelet/pods/9bb12859-245c-4653-a61b-80a45dc8a55c/volumes"
Jun 02 15:57:18.372286 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:18.372256    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:18.372833 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:18.372807    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 15:57:19.143523 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:19.143493    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:57:21.198430 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:21.198382    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:57:28.372907 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:28.372866    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 15:57:38.372885 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:38.372840    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 15:57:48.373548 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:48.373508    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 15:57:56.634887 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.634852    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"]
Jun 02 15:57:56.635369 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.635205    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" containerID="cri-o://0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2" gracePeriod=30
Jun 02 15:57:56.635369 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.635237    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kube-rbac-proxy" containerID="cri-o://eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec" gracePeriod=30
Jun 02 15:57:56.771630 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.771593    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"]
Jun 02 15:57:56.772032 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.772017    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kube-rbac-proxy"
Jun 02 15:57:56.772078 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.772035    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kube-rbac-proxy"
Jun 02 15:57:56.772078 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.772049    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container"
Jun 02 15:57:56.772078 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.772057    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container"
Jun 02 15:57:56.772192 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.772153    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kube-rbac-proxy"
Jun 02 15:57:56.772192 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.772166    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="9bb12859-245c-4653-a61b-80a45dc8a55c" containerName="kserve-container"
Jun 02 15:57:56.774565 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.774551    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:56.776749 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.776725    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f26f3-predictor-serving-cert\""
Jun 02 15:57:56.776858 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.776735    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f26f3-kube-rbac-proxy-sar-config\""
Jun 02 15:57:56.785761 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.785739    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"]
Jun 02 15:57:56.845849 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.845815    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"]
Jun 02 15:57:56.846166 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.846124    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container" containerID="cri-o://0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374" gracePeriod=30
Jun 02 15:57:56.846258 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.846169    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kube-rbac-proxy" containerID="cri-o://4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907" gracePeriod=30
Jun 02 15:57:56.904038 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.904001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-f26f3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-error-404-isvc-f26f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:56.904038 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.904040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l429k\" (UniqueName: \"kubernetes.io/projected/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-kube-api-access-l429k\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:56.904230 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:56.904079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-proxy-tls\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.005104 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.005069    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-f26f3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-error-404-isvc-f26f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.005278 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.005178    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l429k\" (UniqueName: \"kubernetes.io/projected/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-kube-api-access-l429k\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.005278 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.005238    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-proxy-tls\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.005709 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.005691    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-f26f3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-error-404-isvc-f26f3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.007731 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.007711    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-proxy-tls\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.022833 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.022800    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l429k\" (UniqueName: \"kubernetes.io/projected/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-kube-api-access-l429k\") pod \"error-404-isvc-f26f3-predictor-7d74978d64-x6tkp\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") " pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.086066 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.086013    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.215108 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.215060    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"]
Jun 02 15:57:57.217894 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:57:57.217865    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a6872d_9b48_4d8b_8e7e_5c2276f1d377.slice/crio-a1475207f0e548878abce133cd999a4db367ebebe6f297d9ea5264b9cedccc6c WatchSource:0}: Error finding container a1475207f0e548878abce133cd999a4db367ebebe6f297d9ea5264b9cedccc6c: Status 404 returned error can't find the container with id a1475207f0e548878abce133cd999a4db367ebebe6f297d9ea5264b9cedccc6c
Jun 02 15:57:57.504786 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.504703    2577 generic.go:358] "Generic (PLEG): container finished" podID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerID="eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec" exitCode=2
Jun 02 15:57:57.504940 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.504780    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" event={"ID":"42ad842b-0ad1-463a-86ee-a4e55a5e9eba","Type":"ContainerDied","Data":"eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec"}
Jun 02 15:57:57.506429 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.506403    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" event={"ID":"42a6872d-9b48-4d8b-8e7e-5c2276f1d377","Type":"ContainerStarted","Data":"a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75"}
Jun 02 15:57:57.506567 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.506436    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" event={"ID":"42a6872d-9b48-4d8b-8e7e-5c2276f1d377","Type":"ContainerStarted","Data":"8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c"}
Jun 02 15:57:57.506567 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.506450    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" event={"ID":"42a6872d-9b48-4d8b-8e7e-5c2276f1d377","Type":"ContainerStarted","Data":"a1475207f0e548878abce133cd999a4db367ebebe6f297d9ea5264b9cedccc6c"}
Jun 02 15:57:57.506567 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.506509    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:57.508279 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.508258    2577 generic.go:358] "Generic (PLEG): container finished" podID="36beb25a-880a-4154-9112-9be2c2456cc1" containerID="4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907" exitCode=2
Jun 02 15:57:57.508407 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.508295    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" event={"ID":"36beb25a-880a-4154-9112-9be2c2456cc1","Type":"ContainerDied","Data":"4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907"}
Jun 02 15:57:57.527028 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:57.526988    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podStartSLOduration=1.526975321 podStartE2EDuration="1.526975321s" podCreationTimestamp="2026-06-02 15:57:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:57:57.524421883 +0000 UTC m=+681.866843105" watchObservedRunningTime="2026-06-02 15:57:57.526975321 +0000 UTC m=+681.869396593"
Jun 02 15:57:58.373402 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:58.373363    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 15:57:58.511872 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:58.511842    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:57:58.513247 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:58.513218    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 15:57:59.137577 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:59.137540    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jun 02 15:57:59.142956 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:59.142927    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jun 02 15:57:59.514722 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:57:59.514685    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 15:58:00.792981 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.792957    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:58:00.939687 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.939651    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46kq7\" (UniqueName: \"kubernetes.io/projected/36beb25a-880a-4154-9112-9be2c2456cc1-kube-api-access-46kq7\") pod \"36beb25a-880a-4154-9112-9be2c2456cc1\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") "
Jun 02 15:58:00.939687 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.939694    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36beb25a-880a-4154-9112-9be2c2456cc1-kserve-provision-location\") pod \"36beb25a-880a-4154-9112-9be2c2456cc1\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") "
Jun 02 15:58:00.939930 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.939727    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36beb25a-880a-4154-9112-9be2c2456cc1-proxy-tls\") pod \"36beb25a-880a-4154-9112-9be2c2456cc1\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") "
Jun 02 15:58:00.939930 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.939877    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36beb25a-880a-4154-9112-9be2c2456cc1-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") pod \"36beb25a-880a-4154-9112-9be2c2456cc1\" (UID: \"36beb25a-880a-4154-9112-9be2c2456cc1\") "
Jun 02 15:58:00.940116 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.940085    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36beb25a-880a-4154-9112-9be2c2456cc1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "36beb25a-880a-4154-9112-9be2c2456cc1" (UID: "36beb25a-880a-4154-9112-9be2c2456cc1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:58:00.940194 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.940177    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/36beb25a-880a-4154-9112-9be2c2456cc1-kserve-provision-location\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:58:00.940242 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.940170    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36beb25a-880a-4154-9112-9be2c2456cc1-isvc-xgboost-graph-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-kube-rbac-proxy-sar-config") pod "36beb25a-880a-4154-9112-9be2c2456cc1" (UID: "36beb25a-880a-4154-9112-9be2c2456cc1"). InnerVolumeSpecName "isvc-xgboost-graph-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:58:00.941756 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.941739    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36beb25a-880a-4154-9112-9be2c2456cc1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36beb25a-880a-4154-9112-9be2c2456cc1" (UID: "36beb25a-880a-4154-9112-9be2c2456cc1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:58:00.941815 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:00.941753    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36beb25a-880a-4154-9112-9be2c2456cc1-kube-api-access-46kq7" (OuterVolumeSpecName: "kube-api-access-46kq7") pod "36beb25a-880a-4154-9112-9be2c2456cc1" (UID: "36beb25a-880a-4154-9112-9be2c2456cc1"). InnerVolumeSpecName "kube-api-access-46kq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:58:01.040638 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.040599    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46kq7\" (UniqueName: \"kubernetes.io/projected/36beb25a-880a-4154-9112-9be2c2456cc1-kube-api-access-46kq7\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:58:01.040638 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.040632    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36beb25a-880a-4154-9112-9be2c2456cc1-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:58:01.040638 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.040644    2577 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36beb25a-880a-4154-9112-9be2c2456cc1-isvc-xgboost-graph-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:58:01.263749 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.263726    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:58:01.344169 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.344132    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kserve-provision-location\") pod \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") "
Jun 02 15:58:01.344361 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.344206    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-proxy-tls\") pod \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") "
Jun 02 15:58:01.344361 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.344253    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") pod \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") "
Jun 02 15:58:01.344361 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.344287    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxmwj\" (UniqueName: \"kubernetes.io/projected/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kube-api-access-pxmwj\") pod \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\" (UID: \"42ad842b-0ad1-463a-86ee-a4e55a5e9eba\") "
Jun 02 15:58:01.344575 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.344545    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42ad842b-0ad1-463a-86ee-a4e55a5e9eba" (UID: "42ad842b-0ad1-463a-86ee-a4e55a5e9eba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 15:58:01.344654 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.344629    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config") pod "42ad842b-0ad1-463a-86ee-a4e55a5e9eba" (UID: "42ad842b-0ad1-463a-86ee-a4e55a5e9eba"). InnerVolumeSpecName "isvc-sklearn-graph-1-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 15:58:01.344654 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.344647    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kserve-provision-location\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:58:01.346273 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.346253    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42ad842b-0ad1-463a-86ee-a4e55a5e9eba" (UID: "42ad842b-0ad1-463a-86ee-a4e55a5e9eba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 15:58:01.346354 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.346284    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kube-api-access-pxmwj" (OuterVolumeSpecName: "kube-api-access-pxmwj") pod "42ad842b-0ad1-463a-86ee-a4e55a5e9eba" (UID: "42ad842b-0ad1-463a-86ee-a4e55a5e9eba"). InnerVolumeSpecName "kube-api-access-pxmwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 15:58:01.445298 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.445265    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:58:01.445298 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.445296    2577 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:58:01.445298 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.445309    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxmwj\" (UniqueName: \"kubernetes.io/projected/42ad842b-0ad1-463a-86ee-a4e55a5e9eba-kube-api-access-pxmwj\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 15:58:01.522587 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.522505    2577 generic.go:358] "Generic (PLEG): container finished" podID="36beb25a-880a-4154-9112-9be2c2456cc1" containerID="0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374" exitCode=0
Jun 02 15:58:01.522587 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.522584    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"
Jun 02 15:58:01.522781 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.522598    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" event={"ID":"36beb25a-880a-4154-9112-9be2c2456cc1","Type":"ContainerDied","Data":"0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374"}
Jun 02 15:58:01.522781 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.522621    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc" event={"ID":"36beb25a-880a-4154-9112-9be2c2456cc1","Type":"ContainerDied","Data":"c1502b87f1156ecf5dcde398a126dd18dc4d7bfdbc7c4e34d55b5b27863c8edb"}
Jun 02 15:58:01.522781 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.522637    2577 scope.go:117] "RemoveContainer" containerID="4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907"
Jun 02 15:58:01.524496 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.524465    2577 generic.go:358] "Generic (PLEG): container finished" podID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerID="0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2" exitCode=0
Jun 02 15:58:01.524633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.524545    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"
Jun 02 15:58:01.524633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.524545    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" event={"ID":"42ad842b-0ad1-463a-86ee-a4e55a5e9eba","Type":"ContainerDied","Data":"0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2"}
Jun 02 15:58:01.524633 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.524586    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524" event={"ID":"42ad842b-0ad1-463a-86ee-a4e55a5e9eba","Type":"ContainerDied","Data":"53afeaf2593d8a2f17a60e6adc31bddb9d14abd946ac6e85fa09ff9a1939ebc5"}
Jun 02 15:58:01.532418 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.532398    2577 scope.go:117] "RemoveContainer" containerID="0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374"
Jun 02 15:58:01.540443 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.540382    2577 scope.go:117] "RemoveContainer" containerID="070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3"
Jun 02 15:58:01.547911 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.547891    2577 scope.go:117] "RemoveContainer" containerID="4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907"
Jun 02 15:58:01.548195 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:01.548178    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907\": container with ID starting with 4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907 not found: ID does not exist" containerID="4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907"
Jun 02 15:58:01.548237 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.548203    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907"} err="failed to get container status \"4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907\": rpc error: code = NotFound desc = could not find container \"4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907\": container with ID starting with 4e3aedd06cfcd0ef43308ac13ec86f74df4084a719f6455b77b15ca0177e3907 not found: ID does not exist"
Jun 02 15:58:01.548237 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.548231    2577 scope.go:117] "RemoveContainer" containerID="0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374"
Jun 02 15:58:01.548499 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:01.548479    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374\": container with ID starting with 0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374 not found: ID does not exist" containerID="0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374"
Jun 02 15:58:01.548562 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.548505    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374"} err="failed to get container status \"0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374\": rpc error: code = NotFound desc = could not find container \"0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374\": container with ID starting with 0bb74cf123ed5882977c895d5e46000cbf7396d585f07e0c776f2ad3b677b374 not found: ID does not exist"
Jun 02 15:58:01.548562 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.548522    2577 scope.go:117] "RemoveContainer" containerID="070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3"
Jun 02 15:58:01.548756 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:01.548738    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3\": container with ID starting with 070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3 not found: ID does not exist" containerID="070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3"
Jun 02 15:58:01.548799 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.548762    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3"} err="failed to get container status \"070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3\": rpc error: code = NotFound desc = could not find container \"070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3\": container with ID starting with 070654f92382b69a5457eb62bd5b6c44d142b48b037e5205d9ddaa3cf29ef3b3 not found: ID does not exist"
Jun 02 15:58:01.548799 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.548778    2577 scope.go:117] "RemoveContainer" containerID="eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec"
Jun 02 15:58:01.552792 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.552767    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"]
Jun 02 15:58:01.555585 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.555539    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-669d8d6456-svbpc"]
Jun 02 15:58:01.557173 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.557160    2577 scope.go:117] "RemoveContainer" containerID="0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2"
Jun 02 15:58:01.564450 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.564432    2577 scope.go:117] "RemoveContainer" containerID="3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3"
Jun 02 15:58:01.571603 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.571585    2577 scope.go:117] "RemoveContainer" containerID="eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec"
Jun 02 15:58:01.571877 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:01.571861    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec\": container with ID starting with eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec not found: ID does not exist" containerID="eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec"
Jun 02 15:58:01.571943 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.571883    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec"} err="failed to get container status \"eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec\": rpc error: code = NotFound desc = could not find container \"eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec\": container with ID starting with eb5b404b5cb1ec18de8aec4bbdfd7eb820a26e4edab468d48c36df53c8ce22ec not found: ID does not exist"
Jun 02 15:58:01.571943 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.571905    2577 scope.go:117] "RemoveContainer" containerID="0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2"
Jun 02 15:58:01.572195 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:01.572172    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2\": container with ID starting with 0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2 not found: ID does not exist" containerID="0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2"
Jun 02 15:58:01.572266 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.572205    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2"} err="failed to get container status \"0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2\": rpc error: code = NotFound desc = could not find container \"0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2\": container with ID starting with 0863aa07945da4701477921cc1ea6a41fe5452ca70b409d84c655da588a99bc2 not found: ID does not exist"
Jun 02 15:58:01.572350 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.572230    2577 scope.go:117] "RemoveContainer" containerID="3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3"
Jun 02 15:58:01.572768 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:01.572746    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3\": container with ID starting with 3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3 not found: ID does not exist" containerID="3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3"
Jun 02 15:58:01.572877 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.572771    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3"} err="failed to get container status \"3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3\": rpc error: code = NotFound desc = could not find container \"3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3\": container with ID starting with 3803661f65bfe9a2c666ce83d1c68e9e2ea4d232eeae7389450c55dcba9ec2e3 not found: ID does not exist"
Jun 02 15:58:01.574549 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.574529    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"]
Jun 02 15:58:01.581291 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:01.581269    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-55cb655557-fr524"]
Jun 02 15:58:02.223572 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:02.223542    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" path="/var/lib/kubelet/pods/36beb25a-880a-4154-9112-9be2c2456cc1/volumes"
Jun 02 15:58:02.224035 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:02.224022    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" path="/var/lib/kubelet/pods/42ad842b-0ad1-463a-86ee-a4e55a5e9eba/volumes"
Jun 02 15:58:04.518700 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:04.518671    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:58:04.519198 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:04.519173    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 15:58:10.611766 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.611733    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"]
Jun 02 15:58:10.612118 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612058    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kube-rbac-proxy"
Jun 02 15:58:10.612118 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612069    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kube-rbac-proxy"
Jun 02 15:58:10.612118 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612086    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="storage-initializer"
Jun 02 15:58:10.612118 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612091    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="storage-initializer"
Jun 02 15:58:10.612118 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612104    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container"
Jun 02 15:58:10.612118 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612110    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container"
Jun 02 15:58:10.612118 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612121    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="storage-initializer"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612127    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="storage-initializer"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612133    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612138    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612146    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kube-rbac-proxy"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612151    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kube-rbac-proxy"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612200    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kube-rbac-proxy"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612208    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kube-rbac-proxy"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612215    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="42ad842b-0ad1-463a-86ee-a4e55a5e9eba" containerName="kserve-container"
Jun 02 15:58:10.612335 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.612223    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="36beb25a-880a-4154-9112-9be2c2456cc1" containerName="kserve-container"
Jun 02 15:58:10.616498 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.616480    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:10.618534 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.618507    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c6170-serving-cert\""
Jun 02 15:58:10.618748 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.618733    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-c6170-kube-rbac-proxy-sar-config\""
Jun 02 15:58:10.627934 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.627903    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"]
Jun 02 15:58:10.733170 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.733129    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2e86081-416e-496d-9c46-f8e64e64b90f-openshift-service-ca-bundle\") pod \"switch-graph-c6170-7d9d99d4d4-v4zn6\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") " pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:10.733347 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.733178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls\") pod \"switch-graph-c6170-7d9d99d4d4-v4zn6\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") " pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:10.833673 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.833638    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2e86081-416e-496d-9c46-f8e64e64b90f-openshift-service-ca-bundle\") pod \"switch-graph-c6170-7d9d99d4d4-v4zn6\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") " pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:10.833846 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.833689    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls\") pod \"switch-graph-c6170-7d9d99d4d4-v4zn6\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") " pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:10.833846 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:10.833820    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-c6170-serving-cert: secret "switch-graph-c6170-serving-cert" not found
Jun 02 15:58:10.833920 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:10.833905    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls podName:c2e86081-416e-496d-9c46-f8e64e64b90f nodeName:}" failed. No retries permitted until 2026-06-02 15:58:11.333882764 +0000 UTC m=+695.676303985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls") pod "switch-graph-c6170-7d9d99d4d4-v4zn6" (UID: "c2e86081-416e-496d-9c46-f8e64e64b90f") : secret "switch-graph-c6170-serving-cert" not found
Jun 02 15:58:10.834381 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:10.834361    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2e86081-416e-496d-9c46-f8e64e64b90f-openshift-service-ca-bundle\") pod \"switch-graph-c6170-7d9d99d4d4-v4zn6\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") " pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:11.338926 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:11.338893    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls\") pod \"switch-graph-c6170-7d9d99d4d4-v4zn6\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") " pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:11.341319 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:11.341296    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls\") pod \"switch-graph-c6170-7d9d99d4d4-v4zn6\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") " pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:11.531806 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:11.531767    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:11.666838 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:11.666809    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"]
Jun 02 15:58:11.670456 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:58:11.670428    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e86081_416e_496d_9c46_f8e64e64b90f.slice/crio-a4d0b87efbd8f8aa1eeec9a4e69e2bb4722b7a285c718d8763808869f1b462fb WatchSource:0}: Error finding container a4d0b87efbd8f8aa1eeec9a4e69e2bb4722b7a285c718d8763808869f1b462fb: Status 404 returned error can't find the container with id a4d0b87efbd8f8aa1eeec9a4e69e2bb4722b7a285c718d8763808869f1b462fb
Jun 02 15:58:12.566529 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:12.566493    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" event={"ID":"c2e86081-416e-496d-9c46-f8e64e64b90f","Type":"ContainerStarted","Data":"a4d0b87efbd8f8aa1eeec9a4e69e2bb4722b7a285c718d8763808869f1b462fb"}
Jun 02 15:58:14.519479 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:14.519441    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 15:58:14.575548 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:14.575515    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" event={"ID":"c2e86081-416e-496d-9c46-f8e64e64b90f","Type":"ContainerStarted","Data":"a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72"}
Jun 02 15:58:14.575718 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:14.575579    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:14.592021 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:14.591945    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" podStartSLOduration=2.692919865 podStartE2EDuration="4.591929186s" podCreationTimestamp="2026-06-02 15:58:10 +0000 UTC" firstStartedPulling="2026-06-02 15:58:11.672339685 +0000 UTC m=+696.014760885" lastFinishedPulling="2026-06-02 15:58:13.571349003 +0000 UTC m=+697.913770206" observedRunningTime="2026-06-02 15:58:14.591296821 +0000 UTC m=+698.933718043" watchObservedRunningTime="2026-06-02 15:58:14.591929186 +0000 UTC m=+698.934350410"
Jun 02 15:58:20.584928 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:20.584895    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 15:58:24.519974 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:24.519927    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 15:58:34.519328 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:34.519233    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 15:58:44.520330 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:44.520299    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 15:58:56.577744 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.577711    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"]
Jun 02 15:58:56.582622 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.582603    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:56.584625 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.584604    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f26f3-serving-cert\""
Jun 02 15:58:56.584731 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.584606    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-f26f3-kube-rbac-proxy-sar-config\""
Jun 02 15:58:56.588192 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.588171    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"]
Jun 02 15:58:56.727519 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.727487    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98f968c8-d373-4228-a15a-2aab80d97999-openshift-service-ca-bundle\") pod \"sequence-graph-f26f3-6c59bfdcd4-vgtln\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") " pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:56.727694 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.727532    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls\") pod \"sequence-graph-f26f3-6c59bfdcd4-vgtln\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") " pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:56.828717 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.828443    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98f968c8-d373-4228-a15a-2aab80d97999-openshift-service-ca-bundle\") pod \"sequence-graph-f26f3-6c59bfdcd4-vgtln\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") " pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:56.829598 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.829570    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98f968c8-d373-4228-a15a-2aab80d97999-openshift-service-ca-bundle\") pod \"sequence-graph-f26f3-6c59bfdcd4-vgtln\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") " pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:56.829771 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:56.829747    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls\") pod \"sequence-graph-f26f3-6c59bfdcd4-vgtln\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") " pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:56.830110 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:56.830085    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-f26f3-serving-cert: secret "sequence-graph-f26f3-serving-cert" not found
Jun 02 15:58:56.830280 ip-10-0-133-168 kubenswrapper[2577]: E0602 15:58:56.830263    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls podName:98f968c8-d373-4228-a15a-2aab80d97999 nodeName:}" failed. No retries permitted until 2026-06-02 15:58:57.330238825 +0000 UTC m=+741.672660027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls") pod "sequence-graph-f26f3-6c59bfdcd4-vgtln" (UID: "98f968c8-d373-4228-a15a-2aab80d97999") : secret "sequence-graph-f26f3-serving-cert" not found
Jun 02 15:58:57.334117 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.334076    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls\") pod \"sequence-graph-f26f3-6c59bfdcd4-vgtln\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") " pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:57.336414 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.336380    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls\") pod \"sequence-graph-f26f3-6c59bfdcd4-vgtln\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") " pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:57.493703 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.493663    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:57.620994 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.620863    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"]
Jun 02 15:58:57.623807 ip-10-0-133-168 kubenswrapper[2577]: W0602 15:58:57.623775    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice/crio-96e045f9c772d38000f3205b8de4f85c68408f002f92d6bb663d3f088993ef01 WatchSource:0}: Error finding container 96e045f9c772d38000f3205b8de4f85c68408f002f92d6bb663d3f088993ef01: Status 404 returned error can't find the container with id 96e045f9c772d38000f3205b8de4f85c68408f002f92d6bb663d3f088993ef01
Jun 02 15:58:57.625708 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.625692    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 15:58:57.714849 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.714818    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" event={"ID":"98f968c8-d373-4228-a15a-2aab80d97999","Type":"ContainerStarted","Data":"2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4"}
Jun 02 15:58:57.714849 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.714852    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" event={"ID":"98f968c8-d373-4228-a15a-2aab80d97999","Type":"ContainerStarted","Data":"96e045f9c772d38000f3205b8de4f85c68408f002f92d6bb663d3f088993ef01"}
Jun 02 15:58:57.715035 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.714879    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 15:58:57.732067 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:58:57.731935    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" podStartSLOduration=1.731887973 podStartE2EDuration="1.731887973s" podCreationTimestamp="2026-06-02 15:58:56 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 15:58:57.731333962 +0000 UTC m=+742.073755184" watchObservedRunningTime="2026-06-02 15:58:57.731887973 +0000 UTC m=+742.074309197"
Jun 02 15:59:03.724578 ip-10-0-133-168 kubenswrapper[2577]: I0602 15:59:03.724547    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 16:06:25.250577 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.250541    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"]
Jun 02 16:06:25.253043 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.250803    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170" containerID="cri-o://a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72" gracePeriod=30
Jun 02 16:06:25.477113 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.477079    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"]
Jun 02 16:06:25.477345 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.477320    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container" containerID="cri-o://1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112" gracePeriod=30
Jun 02 16:06:25.477434 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.477370    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kube-rbac-proxy" containerID="cri-o://3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7" gracePeriod=30
Jun 02 16:06:25.527660 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.524639    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"]
Jun 02 16:06:25.529835 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.529807    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.532251 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.532215    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d60e3-kube-rbac-proxy-sar-config\""
Jun 02 16:06:25.532376 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.532221    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d60e3-predictor-serving-cert\""
Jun 02 16:06:25.536492 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.536468    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"]
Jun 02 16:06:25.583157 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.583121    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:25.593484 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.593455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-d60e3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-error-404-isvc-d60e3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.593632 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.593506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-proxy-tls\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.593632 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.593569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg8pq\" (UniqueName: \"kubernetes.io/projected/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-kube-api-access-pg8pq\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.694376 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.694326    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-d60e3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-error-404-isvc-d60e3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.694575 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.694411    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-proxy-tls\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.694575 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.694466    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pg8pq\" (UniqueName: \"kubernetes.io/projected/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-kube-api-access-pg8pq\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.695147 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.695124    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-d60e3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-error-404-isvc-d60e3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.696951 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.696933    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-proxy-tls\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.703699 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.703673    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg8pq\" (UniqueName: \"kubernetes.io/projected/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-kube-api-access-pg8pq\") pod \"error-404-isvc-d60e3-predictor-84f4999646-pbx7h\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") " pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.841825 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.841733    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:25.979783 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.979758    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"]
Jun 02 16:06:25.982691 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:06:25.982661    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-fac29f8bb59a32e0283674badabcb778e3cf7557392a9414f487ed3c702d3178 WatchSource:0}: Error finding container fac29f8bb59a32e0283674badabcb778e3cf7557392a9414f487ed3c702d3178: Status 404 returned error can't find the container with id fac29f8bb59a32e0283674badabcb778e3cf7557392a9414f487ed3c702d3178
Jun 02 16:06:25.984355 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:25.984335    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:06:26.081344 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:26.081311    2577 generic.go:358] "Generic (PLEG): container finished" podID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerID="3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7" exitCode=2
Jun 02 16:06:26.081475 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:26.081344    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" event={"ID":"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841","Type":"ContainerDied","Data":"3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7"}
Jun 02 16:06:26.082776 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:26.082754    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" event={"ID":"0d58166f-1528-4a7d-8f78-c4411aa8a2c7","Type":"ContainerStarted","Data":"a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478"}
Jun 02 16:06:26.082868 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:26.082779    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" event={"ID":"0d58166f-1528-4a7d-8f78-c4411aa8a2c7","Type":"ContainerStarted","Data":"fac29f8bb59a32e0283674badabcb778e3cf7557392a9414f487ed3c702d3178"}
Jun 02 16:06:27.088828 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:27.088785    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" event={"ID":"0d58166f-1528-4a7d-8f78-c4411aa8a2c7","Type":"ContainerStarted","Data":"997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19"}
Jun 02 16:06:27.089283 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:27.088887    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:27.109244 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:27.109199    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podStartSLOduration=2.109184445 podStartE2EDuration="2.109184445s" podCreationTimestamp="2026-06-02 16:06:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:06:27.107215742 +0000 UTC m=+1191.449636963" watchObservedRunningTime="2026-06-02 16:06:27.109184445 +0000 UTC m=+1191.451605668"
Jun 02 16:06:28.093083 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.093052    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:28.094575 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.094548    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:28.367909 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.367822    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jun 02 16:06:28.372897 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.372870    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jun 02 16:06:28.816891 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.816869    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 16:06:28.924438 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.924380    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-c6170-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-error-404-isvc-c6170-kube-rbac-proxy-sar-config\") pod \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") "
Jun 02 16:06:28.924610 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.924449    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmvdm\" (UniqueName: \"kubernetes.io/projected/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-kube-api-access-gmvdm\") pod \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") "
Jun 02 16:06:28.924610 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.924468    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-proxy-tls\") pod \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\" (UID: \"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841\") "
Jun 02 16:06:28.924834 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.924788    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-error-404-isvc-c6170-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-c6170-kube-rbac-proxy-sar-config") pod "c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" (UID: "c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841"). InnerVolumeSpecName "error-404-isvc-c6170-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:06:28.926599 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.926573    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" (UID: "c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:06:28.926665 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:28.926610    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-kube-api-access-gmvdm" (OuterVolumeSpecName: "kube-api-access-gmvdm") pod "c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" (UID: "c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841"). InnerVolumeSpecName "kube-api-access-gmvdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:06:29.026113 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.026025    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-c6170-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-error-404-isvc-c6170-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:06:29.026113 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.026056    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvdm\" (UniqueName: \"kubernetes.io/projected/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-kube-api-access-gmvdm\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:06:29.026113 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.026066    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:06:29.099773 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.099741    2577 generic.go:358] "Generic (PLEG): container finished" podID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerID="1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112" exitCode=0
Jun 02 16:06:29.100198 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.099809    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"
Jun 02 16:06:29.100198 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.099824    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" event={"ID":"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841","Type":"ContainerDied","Data":"1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112"}
Jun 02 16:06:29.100198 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.099858    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9" event={"ID":"c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841","Type":"ContainerDied","Data":"a777025fec0646e8b6066a61d016d63f550d17ccb36131e2b9e926f435516176"}
Jun 02 16:06:29.100198 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.099878    2577 scope.go:117] "RemoveContainer" containerID="3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7"
Jun 02 16:06:29.100432 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.100316    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:29.108309 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.108293    2577 scope.go:117] "RemoveContainer" containerID="1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112"
Jun 02 16:06:29.115527 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.115505    2577 scope.go:117] "RemoveContainer" containerID="3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7"
Jun 02 16:06:29.115787 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:06:29.115770    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7\": container with ID starting with 3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7 not found: ID does not exist" containerID="3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7"
Jun 02 16:06:29.115861 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.115796    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7"} err="failed to get container status \"3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7\": rpc error: code = NotFound desc = could not find container \"3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7\": container with ID starting with 3f2cba8db3bd5bbc958125a9d4da397eeea29fbb15b7ed9a77a65cf97b643eb7 not found: ID does not exist"
Jun 02 16:06:29.115861 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.115815    2577 scope.go:117] "RemoveContainer" containerID="1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112"
Jun 02 16:06:29.116019 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:06:29.116006    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112\": container with ID starting with 1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112 not found: ID does not exist" containerID="1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112"
Jun 02 16:06:29.116059 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.116021    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112"} err="failed to get container status \"1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112\": rpc error: code = NotFound desc = could not find container \"1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112\": container with ID starting with 1b5b040bbfdc2e782d2a3dd85261b8ac9aafb0801ec6a81a6659bdb31b364112 not found: ID does not exist"
Jun 02 16:06:29.121178 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.121155    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"]
Jun 02 16:06:29.125710 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:29.125689    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c6170-predictor-59b777545b-kglp9"]
Jun 02 16:06:30.223062 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:30.223032    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" path="/var/lib/kubelet/pods/c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841/volumes"
Jun 02 16:06:30.582467 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:30.582365    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:34.105009 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:34.104978    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:06:34.105443 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:34.105417    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:35.582757 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:35.582656    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:35.583166 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:35.582787    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 16:06:40.583350 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:40.583305    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:44.105819 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:44.105769    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:45.582920 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:45.582877    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:50.583576 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:50.583538    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:06:54.105448 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:54.105412    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:06:55.399734 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:55.399711    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 16:06:55.446186 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:55.446155    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2e86081-416e-496d-9c46-f8e64e64b90f-openshift-service-ca-bundle\") pod \"c2e86081-416e-496d-9c46-f8e64e64b90f\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") "
Jun 02 16:06:55.446356 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:55.446203    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls\") pod \"c2e86081-416e-496d-9c46-f8e64e64b90f\" (UID: \"c2e86081-416e-496d-9c46-f8e64e64b90f\") "
Jun 02 16:06:55.446569 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:55.446542    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e86081-416e-496d-9c46-f8e64e64b90f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c2e86081-416e-496d-9c46-f8e64e64b90f" (UID: "c2e86081-416e-496d-9c46-f8e64e64b90f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:06:55.448223 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:55.448204    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c2e86081-416e-496d-9c46-f8e64e64b90f" (UID: "c2e86081-416e-496d-9c46-f8e64e64b90f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:06:55.547060 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:55.546981    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2e86081-416e-496d-9c46-f8e64e64b90f-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:06:55.547060 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:55.547010    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c2e86081-416e-496d-9c46-f8e64e64b90f-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:06:56.182272 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.182234    2577 generic.go:358] "Generic (PLEG): container finished" podID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerID="a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72" exitCode=0
Jun 02 16:06:56.182556 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.182319    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" event={"ID":"c2e86081-416e-496d-9c46-f8e64e64b90f","Type":"ContainerDied","Data":"a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72"}
Jun 02 16:06:56.182556 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.182352    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"
Jun 02 16:06:56.182556 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.182370    2577 scope.go:117] "RemoveContainer" containerID="a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72"
Jun 02 16:06:56.182556 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.182360    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6" event={"ID":"c2e86081-416e-496d-9c46-f8e64e64b90f","Type":"ContainerDied","Data":"a4d0b87efbd8f8aa1eeec9a4e69e2bb4722b7a285c718d8763808869f1b462fb"}
Jun 02 16:06:56.190188 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.190168    2577 scope.go:117] "RemoveContainer" containerID="a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72"
Jun 02 16:06:56.190468 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:06:56.190449    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72\": container with ID starting with a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72 not found: ID does not exist" containerID="a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72"
Jun 02 16:06:56.190552 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.190476    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72"} err="failed to get container status \"a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72\": rpc error: code = NotFound desc = could not find container \"a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72\": container with ID starting with a11a20b19eba879dcce16d548583ab80ec65e97d47ec01aa592e1cd4b0c18b72 not found: ID does not exist"
Jun 02 16:06:56.207006 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.206981    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"]
Jun 02 16:06:56.211493 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.211468    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-c6170-7d9d99d4d4-v4zn6"]
Jun 02 16:06:56.222880 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:06:56.222857    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" path="/var/lib/kubelet/pods/c2e86081-416e-496d-9c46-f8e64e64b90f/volumes"
Jun 02 16:07:04.105710 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:04.105671    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jun 02 16:07:11.420631 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.420590    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"]
Jun 02 16:07:11.421066 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.420830    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3" containerID="cri-o://2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4" gracePeriod=30
Jun 02 16:07:11.801421 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801320    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"]
Jun 02 16:07:11.801717 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801702    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kube-rbac-proxy"
Jun 02 16:07:11.801760 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801719    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kube-rbac-proxy"
Jun 02 16:07:11.801760 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801730    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170"
Jun 02 16:07:11.801760 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801736    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170"
Jun 02 16:07:11.801760 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801751    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container"
Jun 02 16:07:11.801760 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801757    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container"
Jun 02 16:07:11.801906 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801814    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kserve-container"
Jun 02 16:07:11.801906 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801822    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="c1c39ca9-cd23-4430-9d5c-f0fb1a4c3841" containerName="kube-rbac-proxy"
Jun 02 16:07:11.801906 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.801828    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="c2e86081-416e-496d-9c46-f8e64e64b90f" containerName="switch-graph-c6170"
Jun 02 16:07:11.804954 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.804936    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.806834 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.806811    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5da44-predictor-serving-cert\""
Jun 02 16:07:11.807122 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.807096    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-5da44-kube-rbac-proxy-sar-config\""
Jun 02 16:07:11.816459 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.816433    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"]
Jun 02 16:07:11.847099 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.847068    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"]
Jun 02 16:07:11.847357 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.847328    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container" containerID="cri-o://8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c" gracePeriod=30
Jun 02 16:07:11.847465 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.847380    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kube-rbac-proxy" containerID="cri-o://a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75" gracePeriod=30
Jun 02 16:07:11.879050 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.879010    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0990edda-a3df-4e13-ac79-c3601c8849aa-proxy-tls\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.879200 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.879143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-5da44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0990edda-a3df-4e13-ac79-c3601c8849aa-error-404-isvc-5da44-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.879255 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.879194    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g6mx\" (UniqueName: \"kubernetes.io/projected/0990edda-a3df-4e13-ac79-c3601c8849aa-kube-api-access-2g6mx\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.979831 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.979793    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2g6mx\" (UniqueName: \"kubernetes.io/projected/0990edda-a3df-4e13-ac79-c3601c8849aa-kube-api-access-2g6mx\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.979965 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.979879    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0990edda-a3df-4e13-ac79-c3601c8849aa-proxy-tls\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.979965 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.979929    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-5da44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0990edda-a3df-4e13-ac79-c3601c8849aa-error-404-isvc-5da44-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.980613 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.980590    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-5da44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0990edda-a3df-4e13-ac79-c3601c8849aa-error-404-isvc-5da44-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.982317 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.982295    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0990edda-a3df-4e13-ac79-c3601c8849aa-proxy-tls\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:11.988079 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:11.988056    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g6mx\" (UniqueName: \"kubernetes.io/projected/0990edda-a3df-4e13-ac79-c3601c8849aa-kube-api-access-2g6mx\") pod \"error-404-isvc-5da44-predictor-98d5465cd-hvgvx\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") " pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:12.115739 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:12.115650    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:12.237965 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:12.237935    2577 generic.go:358] "Generic (PLEG): container finished" podID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerID="a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75" exitCode=2
Jun 02 16:07:12.238124 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:12.237994    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" event={"ID":"42a6872d-9b48-4d8b-8e7e-5c2276f1d377","Type":"ContainerDied","Data":"a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75"}
Jun 02 16:07:12.248907 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:12.248880    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"]
Jun 02 16:07:12.251233 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:07:12.251205    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0990edda_a3df_4e13_ac79_c3601c8849aa.slice/crio-0c494004d027f1a700afcf26b2289ae7768946875573a0f19a0ab2f6227d4793 WatchSource:0}: Error finding container 0c494004d027f1a700afcf26b2289ae7768946875573a0f19a0ab2f6227d4793: Status 404 returned error can't find the container with id 0c494004d027f1a700afcf26b2289ae7768946875573a0f19a0ab2f6227d4793
Jun 02 16:07:13.242092 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:13.242056    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" event={"ID":"0990edda-a3df-4e13-ac79-c3601c8849aa","Type":"ContainerStarted","Data":"a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd"}
Jun 02 16:07:13.242092 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:13.242091    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" event={"ID":"0990edda-a3df-4e13-ac79-c3601c8849aa","Type":"ContainerStarted","Data":"c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f"}
Jun 02 16:07:13.242092 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:13.242101    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" event={"ID":"0990edda-a3df-4e13-ac79-c3601c8849aa","Type":"ContainerStarted","Data":"0c494004d027f1a700afcf26b2289ae7768946875573a0f19a0ab2f6227d4793"}
Jun 02 16:07:13.242619 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:13.242186    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:13.261931 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:13.261884    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podStartSLOduration=2.261869664 podStartE2EDuration="2.261869664s" podCreationTimestamp="2026-06-02 16:07:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:07:13.260576188 +0000 UTC m=+1237.602997410" watchObservedRunningTime="2026-06-02 16:07:13.261869664 +0000 UTC m=+1237.604290886"
Jun 02 16:07:13.722924 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:13.722886    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:14.106182 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:14.106109    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:07:14.245122 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:14.245092    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:14.246575 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:14.246552    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:14.515721 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:14.515669    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jun 02 16:07:14.519984 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:14.519955    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jun 02 16:07:15.199222 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.199198    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 16:07:15.249336 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.249244    2577 generic.go:358] "Generic (PLEG): container finished" podID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerID="8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c" exitCode=0
Jun 02 16:07:15.249821 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.249330    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" event={"ID":"42a6872d-9b48-4d8b-8e7e-5c2276f1d377","Type":"ContainerDied","Data":"8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c"}
Jun 02 16:07:15.249821 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.249360    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"
Jun 02 16:07:15.249821 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.249379    2577 scope.go:117] "RemoveContainer" containerID="a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75"
Jun 02 16:07:15.249821 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.249367    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp" event={"ID":"42a6872d-9b48-4d8b-8e7e-5c2276f1d377","Type":"ContainerDied","Data":"a1475207f0e548878abce133cd999a4db367ebebe6f297d9ea5264b9cedccc6c"}
Jun 02 16:07:15.250010 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.249873    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:15.257456 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.257436    2577 scope.go:117] "RemoveContainer" containerID="8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c"
Jun 02 16:07:15.265000 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.264982    2577 scope.go:117] "RemoveContainer" containerID="a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75"
Jun 02 16:07:15.265265 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:15.265245    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75\": container with ID starting with a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75 not found: ID does not exist" containerID="a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75"
Jun 02 16:07:15.265311 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.265276    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75"} err="failed to get container status \"a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75\": rpc error: code = NotFound desc = could not find container \"a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75\": container with ID starting with a1436bc10891b38880b0f0f3a202781d5acf7e548c026eb1ff5fa02fd084cb75 not found: ID does not exist"
Jun 02 16:07:15.265311 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.265296    2577 scope.go:117] "RemoveContainer" containerID="8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c"
Jun 02 16:07:15.265566 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:15.265549    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c\": container with ID starting with 8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c not found: ID does not exist" containerID="8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c"
Jun 02 16:07:15.265629 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.265572    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c"} err="failed to get container status \"8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c\": rpc error: code = NotFound desc = could not find container \"8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c\": container with ID starting with 8288463e5c69cefd3f653809083870246c700a25eec98525d055d2ec50db194c not found: ID does not exist"
Jun 02 16:07:15.313054 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.313019    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l429k\" (UniqueName: \"kubernetes.io/projected/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-kube-api-access-l429k\") pod \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") "
Jun 02 16:07:15.313230 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.313127    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-proxy-tls\") pod \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") "
Jun 02 16:07:15.313230 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.313176    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-f26f3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-error-404-isvc-f26f3-kube-rbac-proxy-sar-config\") pod \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\" (UID: \"42a6872d-9b48-4d8b-8e7e-5c2276f1d377\") "
Jun 02 16:07:15.313678 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.313642    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-error-404-isvc-f26f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-f26f3-kube-rbac-proxy-sar-config") pod "42a6872d-9b48-4d8b-8e7e-5c2276f1d377" (UID: "42a6872d-9b48-4d8b-8e7e-5c2276f1d377"). InnerVolumeSpecName "error-404-isvc-f26f3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:07:15.315207 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.315176    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-kube-api-access-l429k" (OuterVolumeSpecName: "kube-api-access-l429k") pod "42a6872d-9b48-4d8b-8e7e-5c2276f1d377" (UID: "42a6872d-9b48-4d8b-8e7e-5c2276f1d377"). InnerVolumeSpecName "kube-api-access-l429k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:07:15.315308 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.315233    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42a6872d-9b48-4d8b-8e7e-5c2276f1d377" (UID: "42a6872d-9b48-4d8b-8e7e-5c2276f1d377"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:07:15.413820 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.413778    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:07:15.413820 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.413812    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-f26f3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-error-404-isvc-f26f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:07:15.413820 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.413826    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l429k\" (UniqueName: \"kubernetes.io/projected/42a6872d-9b48-4d8b-8e7e-5c2276f1d377-kube-api-access-l429k\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:07:15.576033 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.575998    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"]
Jun 02 16:07:15.580978 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:15.580947    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f26f3-predictor-7d74978d64-x6tkp"]
Jun 02 16:07:16.222843 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:16.222808    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" path="/var/lib/kubelet/pods/42a6872d-9b48-4d8b-8e7e-5c2276f1d377/volumes"
Jun 02 16:07:18.722954 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:18.722916    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:20.254266 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:20.254233    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:07:20.254697 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:20.254673    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:23.722734 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:23.722695    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:23.723173 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:23.722840    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 16:07:25.486367 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.486271    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"]
Jun 02 16:07:25.486850 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.486802    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kube-rbac-proxy"
Jun 02 16:07:25.486850 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.486821    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kube-rbac-proxy"
Jun 02 16:07:25.486850 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.486834    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container"
Jun 02 16:07:25.486850 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.486842    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container"
Jun 02 16:07:25.487057 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.486934    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kserve-container"
Jun 02 16:07:25.487057 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.486950    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="42a6872d-9b48-4d8b-8e7e-5c2276f1d377" containerName="kube-rbac-proxy"
Jun 02 16:07:25.491480 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.491460    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:25.493942 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.493922    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d60e3-kube-rbac-proxy-sar-config\""
Jun 02 16:07:25.494049 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.493924    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-d60e3-serving-cert\""
Jun 02 16:07:25.500033 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.500007    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"]
Jun 02 16:07:25.610675 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.610635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bd3dec4-786d-4778-9af4-a189ab900723-openshift-service-ca-bundle\") pod \"ensemble-graph-d60e3-5bd89c6854-mzd9g\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") " pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:25.610675 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.610678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls\") pod \"ensemble-graph-d60e3-5bd89c6854-mzd9g\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") " pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:25.712295 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.712256    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bd3dec4-786d-4778-9af4-a189ab900723-openshift-service-ca-bundle\") pod \"ensemble-graph-d60e3-5bd89c6854-mzd9g\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") " pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:25.712295 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.712296    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls\") pod \"ensemble-graph-d60e3-5bd89c6854-mzd9g\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") " pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:25.712536 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:25.712455    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-d60e3-serving-cert: secret "ensemble-graph-d60e3-serving-cert" not found
Jun 02 16:07:25.712536 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:25.712527    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls podName:4bd3dec4-786d-4778-9af4-a189ab900723 nodeName:}" failed. No retries permitted until 2026-06-02 16:07:26.212504852 +0000 UTC m=+1250.554926052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls") pod "ensemble-graph-d60e3-5bd89c6854-mzd9g" (UID: "4bd3dec4-786d-4778-9af4-a189ab900723") : secret "ensemble-graph-d60e3-serving-cert" not found
Jun 02 16:07:25.713031 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:25.713010    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bd3dec4-786d-4778-9af4-a189ab900723-openshift-service-ca-bundle\") pod \"ensemble-graph-d60e3-5bd89c6854-mzd9g\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") " pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:26.217308 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:26.217273    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls\") pod \"ensemble-graph-d60e3-5bd89c6854-mzd9g\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") " pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:26.219942 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:26.219910    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls\") pod \"ensemble-graph-d60e3-5bd89c6854-mzd9g\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") " pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:26.402701 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:26.402665    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:26.528831 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:26.528779    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"]
Jun 02 16:07:26.532609 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:07:26.532579    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd3dec4_786d_4778_9af4_a189ab900723.slice/crio-51316ddcacbd4f93bfde415b40d88d7f2bce2bf09afdf11fd1a6144b3d87d306 WatchSource:0}: Error finding container 51316ddcacbd4f93bfde415b40d88d7f2bce2bf09afdf11fd1a6144b3d87d306: Status 404 returned error can't find the container with id 51316ddcacbd4f93bfde415b40d88d7f2bce2bf09afdf11fd1a6144b3d87d306
Jun 02 16:07:27.300424 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:27.300376    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" event={"ID":"4bd3dec4-786d-4778-9af4-a189ab900723","Type":"ContainerStarted","Data":"16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b"}
Jun 02 16:07:27.300424 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:27.300425    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" event={"ID":"4bd3dec4-786d-4778-9af4-a189ab900723","Type":"ContainerStarted","Data":"51316ddcacbd4f93bfde415b40d88d7f2bce2bf09afdf11fd1a6144b3d87d306"}
Jun 02 16:07:27.300714 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:27.300535    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:27.317534 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:27.317275    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" podStartSLOduration=2.317259742 podStartE2EDuration="2.317259742s" podCreationTimestamp="2026-06-02 16:07:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:07:27.316722537 +0000 UTC m=+1251.659143759" watchObservedRunningTime="2026-06-02 16:07:27.317259742 +0000 UTC m=+1251.659680963"
Jun 02 16:07:28.722691 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:28.722648    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:30.254641 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:30.254605    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:33.309213 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:33.309181    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:33.722027 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:33.721987    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:35.566298 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.566217    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"]
Jun 02 16:07:35.566679 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.566460    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3" containerID="cri-o://16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b" gracePeriod=30
Jun 02 16:07:35.835952 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.835864    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"]
Jun 02 16:07:35.836282 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.836248    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container" containerID="cri-o://a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478" gracePeriod=30
Jun 02 16:07:35.836369 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.836325    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kube-rbac-proxy" containerID="cri-o://997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19" gracePeriod=30
Jun 02 16:07:35.896182 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.896149    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"]
Jun 02 16:07:35.899721 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.899705    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:35.901967 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.901946    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-94f66-predictor-serving-cert\""
Jun 02 16:07:35.902084 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.901948    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-94f66-kube-rbac-proxy-sar-config\""
Jun 02 16:07:35.908728 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.908705    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"]
Jun 02 16:07:35.910045 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.910025    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:35.910122 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.910087    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-94f66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49dd9b8f-7879-42a9-9817-fee62e3d47fd-error-404-isvc-94f66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:35.910162 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:35.910121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vq8\" (UniqueName: \"kubernetes.io/projected/49dd9b8f-7879-42a9-9817-fee62e3d47fd-kube-api-access-q9vq8\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.010801 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.010746    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-94f66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49dd9b8f-7879-42a9-9817-fee62e3d47fd-error-404-isvc-94f66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.010990 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.010828    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vq8\" (UniqueName: \"kubernetes.io/projected/49dd9b8f-7879-42a9-9817-fee62e3d47fd-kube-api-access-q9vq8\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.010990 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.010885    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.011073 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:36.010995    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-94f66-predictor-serving-cert: secret "error-404-isvc-94f66-predictor-serving-cert" not found
Jun 02 16:07:36.011073 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:36.011058    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls podName:49dd9b8f-7879-42a9-9817-fee62e3d47fd nodeName:}" failed. No retries permitted until 2026-06-02 16:07:36.511038803 +0000 UTC m=+1260.853460004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls") pod "error-404-isvc-94f66-predictor-76446b9648-x8kt2" (UID: "49dd9b8f-7879-42a9-9817-fee62e3d47fd") : secret "error-404-isvc-94f66-predictor-serving-cert" not found
Jun 02 16:07:36.011451 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.011432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-94f66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49dd9b8f-7879-42a9-9817-fee62e3d47fd-error-404-isvc-94f66-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.019587 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.019564    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vq8\" (UniqueName: \"kubernetes.io/projected/49dd9b8f-7879-42a9-9817-fee62e3d47fd-kube-api-access-q9vq8\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.331990 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.331955    2577 generic.go:358] "Generic (PLEG): container finished" podID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerID="997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19" exitCode=2
Jun 02 16:07:36.332146 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.332004    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" event={"ID":"0d58166f-1528-4a7d-8f78-c4411aa8a2c7","Type":"ContainerDied","Data":"997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19"}
Jun 02 16:07:36.515442 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.515306    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.517775 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.517745    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls\") pod \"error-404-isvc-94f66-predictor-76446b9648-x8kt2\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") " pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.811072 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.810983    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:36.946290 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:36.946167    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"]
Jun 02 16:07:37.336987 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:37.336905    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" event={"ID":"49dd9b8f-7879-42a9-9817-fee62e3d47fd","Type":"ContainerStarted","Data":"6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558"}
Jun 02 16:07:37.336987 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:37.336942    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" event={"ID":"49dd9b8f-7879-42a9-9817-fee62e3d47fd","Type":"ContainerStarted","Data":"f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a"}
Jun 02 16:07:37.336987 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:37.336952    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" event={"ID":"49dd9b8f-7879-42a9-9817-fee62e3d47fd","Type":"ContainerStarted","Data":"012abfa9fd1fc137693a4f4d8ba1c52ef214eaebf89b994c50464363f2eee443"}
Jun 02 16:07:37.337209 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:37.337064    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:37.356135 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:37.356090    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podStartSLOduration=2.3560766969999998 podStartE2EDuration="2.356076697s" podCreationTimestamp="2026-06-02 16:07:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:07:37.353205874 +0000 UTC m=+1261.695627096" watchObservedRunningTime="2026-06-02 16:07:37.356076697 +0000 UTC m=+1261.698497919"
Jun 02 16:07:38.307791 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:38.307757    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:38.340032 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:38.340003    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:38.341196 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:38.341169    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:07:38.722740 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:38.722700    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:39.076853 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.076831    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:07:39.134365 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.134333    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-d60e3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-error-404-isvc-d60e3-kube-rbac-proxy-sar-config\") pod \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") "
Jun 02 16:07:39.134531 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.134438    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pg8pq\" (UniqueName: \"kubernetes.io/projected/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-kube-api-access-pg8pq\") pod \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") "
Jun 02 16:07:39.134531 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.134463    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-proxy-tls\") pod \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\" (UID: \"0d58166f-1528-4a7d-8f78-c4411aa8a2c7\") "
Jun 02 16:07:39.134738 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.134704    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-error-404-isvc-d60e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-d60e3-kube-rbac-proxy-sar-config") pod "0d58166f-1528-4a7d-8f78-c4411aa8a2c7" (UID: "0d58166f-1528-4a7d-8f78-c4411aa8a2c7"). InnerVolumeSpecName "error-404-isvc-d60e3-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:07:39.136520 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.136495    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-kube-api-access-pg8pq" (OuterVolumeSpecName: "kube-api-access-pg8pq") pod "0d58166f-1528-4a7d-8f78-c4411aa8a2c7" (UID: "0d58166f-1528-4a7d-8f78-c4411aa8a2c7"). InnerVolumeSpecName "kube-api-access-pg8pq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:07:39.136591 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.136511    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d58166f-1528-4a7d-8f78-c4411aa8a2c7" (UID: "0d58166f-1528-4a7d-8f78-c4411aa8a2c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:07:39.235374 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.235337    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-d60e3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-error-404-isvc-d60e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:07:39.235374 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.235370    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg8pq\" (UniqueName: \"kubernetes.io/projected/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-kube-api-access-pg8pq\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:07:39.235374 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.235380    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d58166f-1528-4a7d-8f78-c4411aa8a2c7-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:07:39.344097 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.344010    2577 generic.go:358] "Generic (PLEG): container finished" podID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerID="a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478" exitCode=0
Jun 02 16:07:39.344097 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.344090    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"
Jun 02 16:07:39.344596 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.344095    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" event={"ID":"0d58166f-1528-4a7d-8f78-c4411aa8a2c7","Type":"ContainerDied","Data":"a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478"}
Jun 02 16:07:39.344596 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.344135    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h" event={"ID":"0d58166f-1528-4a7d-8f78-c4411aa8a2c7","Type":"ContainerDied","Data":"fac29f8bb59a32e0283674badabcb778e3cf7557392a9414f487ed3c702d3178"}
Jun 02 16:07:39.344596 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.344151    2577 scope.go:117] "RemoveContainer" containerID="997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19"
Jun 02 16:07:39.344767 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.344739    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:07:39.353011 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.352986    2577 scope.go:117] "RemoveContainer" containerID="a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478"
Jun 02 16:07:39.359865 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.359849    2577 scope.go:117] "RemoveContainer" containerID="997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19"
Jun 02 16:07:39.360118 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:39.360101    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19\": container with ID starting with 997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19 not found: ID does not exist" containerID="997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19"
Jun 02 16:07:39.360175 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.360126    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19"} err="failed to get container status \"997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19\": rpc error: code = NotFound desc = could not find container \"997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19\": container with ID starting with 997bbcd0b0e01c3a8b9d2b159e0d5ea612e35c7e93049af9ef5219b568521b19 not found: ID does not exist"
Jun 02 16:07:39.360175 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.360144    2577 scope.go:117] "RemoveContainer" containerID="a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478"
Jun 02 16:07:39.360423 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:39.360406    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478\": container with ID starting with a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478 not found: ID does not exist" containerID="a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478"
Jun 02 16:07:39.360481 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.360425    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478"} err="failed to get container status \"a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478\": rpc error: code = NotFound desc = could not find container \"a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478\": container with ID starting with a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478 not found: ID does not exist"
Jun 02 16:07:39.367646 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.367623    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"]
Jun 02 16:07:39.374233 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:39.374201    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d60e3-predictor-84f4999646-pbx7h"]
Jun 02 16:07:40.225022 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:40.224982    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" path="/var/lib/kubelet/pods/0d58166f-1528-4a7d-8f78-c4411aa8a2c7/volumes"
Jun 02 16:07:40.254900 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:40.254862    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:42.056939 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.056914    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 16:07:42.157550 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.157516    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls\") pod \"98f968c8-d373-4228-a15a-2aab80d97999\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") "
Jun 02 16:07:42.157739 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.157624    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98f968c8-d373-4228-a15a-2aab80d97999-openshift-service-ca-bundle\") pod \"98f968c8-d373-4228-a15a-2aab80d97999\" (UID: \"98f968c8-d373-4228-a15a-2aab80d97999\") "
Jun 02 16:07:42.157966 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.157943    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f968c8-d373-4228-a15a-2aab80d97999-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "98f968c8-d373-4228-a15a-2aab80d97999" (UID: "98f968c8-d373-4228-a15a-2aab80d97999"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:07:42.159655 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.159632    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98f968c8-d373-4228-a15a-2aab80d97999" (UID: "98f968c8-d373-4228-a15a-2aab80d97999"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:07:42.258815 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.258783    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98f968c8-d373-4228-a15a-2aab80d97999-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:07:42.258815 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.258811    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f968c8-d373-4228-a15a-2aab80d97999-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:07:42.355672 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.355637    2577 generic.go:358] "Generic (PLEG): container finished" podID="98f968c8-d373-4228-a15a-2aab80d97999" containerID="2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4" exitCode=0
Jun 02 16:07:42.355872 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.355693    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" event={"ID":"98f968c8-d373-4228-a15a-2aab80d97999","Type":"ContainerDied","Data":"2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4"}
Jun 02 16:07:42.355872 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.355702    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"
Jun 02 16:07:42.355872 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.355719    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln" event={"ID":"98f968c8-d373-4228-a15a-2aab80d97999","Type":"ContainerDied","Data":"96e045f9c772d38000f3205b8de4f85c68408f002f92d6bb663d3f088993ef01"}
Jun 02 16:07:42.355872 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.355733    2577 scope.go:117] "RemoveContainer" containerID="2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4"
Jun 02 16:07:42.364002 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.363985    2577 scope.go:117] "RemoveContainer" containerID="2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4"
Jun 02 16:07:42.364233 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:42.364217    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4\": container with ID starting with 2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4 not found: ID does not exist" containerID="2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4"
Jun 02 16:07:42.364280 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.364240    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4"} err="failed to get container status \"2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4\": rpc error: code = NotFound desc = could not find container \"2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4\": container with ID starting with 2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4 not found: ID does not exist"
Jun 02 16:07:42.372484 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.372436    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"]
Jun 02 16:07:42.377774 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:42.377749    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-f26f3-6c59bfdcd4-vgtln"]
Jun 02 16:07:43.308569 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:43.308529    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:44.222635 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:44.222604    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f968c8-d373-4228-a15a-2aab80d97999" path="/var/lib/kubelet/pods/98f968c8-d373-4228-a15a-2aab80d97999/volumes"
Jun 02 16:07:44.348916 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:44.348888    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:07:44.349502 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:44.349472    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:07:48.307717 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:48.307677    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:48.308124 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:48.307792    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:07:50.254876 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:50.254833    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jun 02 16:07:53.308562 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:53.308525    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:07:54.350483 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:54.350441    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:07:57.669619 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:07:57.669587    2577 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/7086319dda216e8db609340441c8f4df9df0cce283bdf80cf21261e06df9c219/diff" to get inode usage: stat /var/lib/containers/storage/overlay/7086319dda216e8db609340441c8f4df9df0cce283bdf80cf21261e06df9c219/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/kserve-ci-e2e-test_sequence-graph-f26f3-6c59bfdcd4-vgtln_98f968c8-d373-4228-a15a-2aab80d97999/sequence-graph-f26f3/0.log" to get inode usage: stat /var/log/pods/kserve-ci-e2e-test_sequence-graph-f26f3-6c59bfdcd4-vgtln_98f968c8-d373-4228-a15a-2aab80d97999/sequence-graph-f26f3/0.log: no such file or directory
Jun 02 16:07:58.307858 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:07:58.307822    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:00.256028 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:00.255997    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:08:03.307901 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:03.307850    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:04.349616 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:04.349580    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:08:05.591646 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:05.591588    2577 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd3dec4_786d_4778_9af4_a189ab900723.slice/crio-16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice/crio-96e045f9c772d38000f3205b8de4f85c68408f002f92d6bb663d3f088993ef01\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice/crio-conmon-2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-conmon-a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-fac29f8bb59a32e0283674badabcb778e3cf7557392a9414f487ed3c702d3178\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice\": RecentStats: unable to find data in memory cache]"
Jun 02 16:08:05.592091 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:05.591602    2577 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd3dec4_786d_4778_9af4_a189ab900723.slice/crio-16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-fac29f8bb59a32e0283674badabcb778e3cf7557392a9414f487ed3c702d3178\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice/crio-conmon-2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-conmon-a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice/crio-96e045f9c772d38000f3205b8de4f85c68408f002f92d6bb663d3f088993ef01\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478.scope\": RecentStats: unable to find data in memory cache]"
Jun 02 16:08:05.592174 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:05.592123    2577 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-conmon-a000adcbe8f7652065dc30abded98485b4e1d1d5f2c5fc0a11c28c2b0d0f5478.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice/crio-conmon-2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d58166f_1528_4a7d_8f78_c4411aa8a2c7.slice/crio-fac29f8bb59a32e0283674badabcb778e3cf7557392a9414f487ed3c702d3178\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice/crio-96e045f9c772d38000f3205b8de4f85c68408f002f92d6bb663d3f088993ef01\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd3dec4_786d_4778_9af4_a189ab900723.slice/crio-16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd3dec4_786d_4778_9af4_a189ab900723.slice/crio-conmon-16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f968c8_d373_4228_a15a_2aab80d97999.slice/crio-2d1dd48905cbc58a12447a736bdd211dc428afe92a760f28380765043b2b01e4.scope\": RecentStats: unable to find data in memory cache]"
Jun 02 16:08:05.727462 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:05.727434    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:08:05.870987 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:05.870907    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bd3dec4-786d-4778-9af4-a189ab900723-openshift-service-ca-bundle\") pod \"4bd3dec4-786d-4778-9af4-a189ab900723\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") "
Jun 02 16:08:05.871146 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:05.871009    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls\") pod \"4bd3dec4-786d-4778-9af4-a189ab900723\" (UID: \"4bd3dec4-786d-4778-9af4-a189ab900723\") "
Jun 02 16:08:05.871267 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:05.871243    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bd3dec4-786d-4778-9af4-a189ab900723-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4bd3dec4-786d-4778-9af4-a189ab900723" (UID: "4bd3dec4-786d-4778-9af4-a189ab900723"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:08:05.873064 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:05.873040    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4bd3dec4-786d-4778-9af4-a189ab900723" (UID: "4bd3dec4-786d-4778-9af4-a189ab900723"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:08:05.972317 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:05.972266    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4bd3dec4-786d-4778-9af4-a189ab900723-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:08:05.972317 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:05.972311    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bd3dec4-786d-4778-9af4-a189ab900723-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:08:06.428184 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.428150    2577 generic.go:358] "Generic (PLEG): container finished" podID="4bd3dec4-786d-4778-9af4-a189ab900723" containerID="16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b" exitCode=0
Jun 02 16:08:06.428372 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.428208    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" event={"ID":"4bd3dec4-786d-4778-9af4-a189ab900723","Type":"ContainerDied","Data":"16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b"}
Jun 02 16:08:06.428372 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.428223    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"
Jun 02 16:08:06.428372 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.428239    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g" event={"ID":"4bd3dec4-786d-4778-9af4-a189ab900723","Type":"ContainerDied","Data":"51316ddcacbd4f93bfde415b40d88d7f2bce2bf09afdf11fd1a6144b3d87d306"}
Jun 02 16:08:06.428372 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.428259    2577 scope.go:117] "RemoveContainer" containerID="16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b"
Jun 02 16:08:06.436422 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.436383    2577 scope.go:117] "RemoveContainer" containerID="16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b"
Jun 02 16:08:06.436681 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:06.436663    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b\": container with ID starting with 16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b not found: ID does not exist" containerID="16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b"
Jun 02 16:08:06.436746 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.436690    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b"} err="failed to get container status \"16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b\": rpc error: code = NotFound desc = could not find container \"16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b\": container with ID starting with 16d2ed9e981683d7c84f3547d72b2aaf2aa48c02191cad564d5ce14ba0d46e6b not found: ID does not exist"
Jun 02 16:08:06.449018 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.448987    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"]
Jun 02 16:08:06.455427 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:06.455400    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-d60e3-5bd89c6854-mzd9g"]
Jun 02 16:08:08.222638 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:08.222608    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" path="/var/lib/kubelet/pods/4bd3dec4-786d-4778-9af4-a189ab900723/volumes"
Jun 02 16:08:11.655312 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655275    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"]
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655620    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kube-rbac-proxy"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655630    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kube-rbac-proxy"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655652    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655657    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655668    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655674    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655681    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655687    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655735    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kserve-container"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655744    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="4bd3dec4-786d-4778-9af4-a189ab900723" containerName="ensemble-graph-d60e3"
Jun 02 16:08:11.655750 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655752    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="98f968c8-d373-4228-a15a-2aab80d97999" containerName="sequence-graph-f26f3"
Jun 02 16:08:11.656118 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.655759    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0d58166f-1528-4a7d-8f78-c4411aa8a2c7" containerName="kube-rbac-proxy"
Jun 02 16:08:11.659777 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.659757    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:11.661898 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.661875    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-5da44-kube-rbac-proxy-sar-config\""
Jun 02 16:08:11.661898 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.661892    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-5da44-serving-cert\""
Jun 02 16:08:11.666649 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.666623    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"]
Jun 02 16:08:11.828916 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.828881    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cce28d2-504f-4153-975d-58978a4367c3-openshift-service-ca-bundle\") pod \"sequence-graph-5da44-7795d6f48-6llxp\" (UID: \"7cce28d2-504f-4153-975d-58978a4367c3\") " pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:11.829093 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.828944    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cce28d2-504f-4153-975d-58978a4367c3-proxy-tls\") pod \"sequence-graph-5da44-7795d6f48-6llxp\" (UID: \"7cce28d2-504f-4153-975d-58978a4367c3\") " pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:11.930000 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.929907    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cce28d2-504f-4153-975d-58978a4367c3-openshift-service-ca-bundle\") pod \"sequence-graph-5da44-7795d6f48-6llxp\" (UID: \"7cce28d2-504f-4153-975d-58978a4367c3\") " pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:11.930000 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.929969    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cce28d2-504f-4153-975d-58978a4367c3-proxy-tls\") pod \"sequence-graph-5da44-7795d6f48-6llxp\" (UID: \"7cce28d2-504f-4153-975d-58978a4367c3\") " pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:11.930736 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.930709    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cce28d2-504f-4153-975d-58978a4367c3-openshift-service-ca-bundle\") pod \"sequence-graph-5da44-7795d6f48-6llxp\" (UID: \"7cce28d2-504f-4153-975d-58978a4367c3\") " pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:11.932459 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.932442    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cce28d2-504f-4153-975d-58978a4367c3-proxy-tls\") pod \"sequence-graph-5da44-7795d6f48-6llxp\" (UID: \"7cce28d2-504f-4153-975d-58978a4367c3\") " pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:11.970641 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:11.970598    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:12.100170 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:12.100134    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"]
Jun 02 16:08:12.103549 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:08:12.103518    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cce28d2_504f_4153_975d_58978a4367c3.slice/crio-f34d4b02f2141a1b684341c0fafc4bb1160eeba3263d27392151655d57318b74 WatchSource:0}: Error finding container f34d4b02f2141a1b684341c0fafc4bb1160eeba3263d27392151655d57318b74: Status 404 returned error can't find the container with id f34d4b02f2141a1b684341c0fafc4bb1160eeba3263d27392151655d57318b74
Jun 02 16:08:12.449551 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:12.449505    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" event={"ID":"7cce28d2-504f-4153-975d-58978a4367c3","Type":"ContainerStarted","Data":"f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c"}
Jun 02 16:08:12.449551 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:12.449546    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" event={"ID":"7cce28d2-504f-4153-975d-58978a4367c3","Type":"ContainerStarted","Data":"f34d4b02f2141a1b684341c0fafc4bb1160eeba3263d27392151655d57318b74"}
Jun 02 16:08:12.449801 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:12.449620    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:12.469097 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:12.469032    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" podStartSLOduration=1.469010999 podStartE2EDuration="1.469010999s" podCreationTimestamp="2026-06-02 16:08:11 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:08:12.467725747 +0000 UTC m=+1296.810146970" watchObservedRunningTime="2026-06-02 16:08:12.469010999 +0000 UTC m=+1296.811432222"
Jun 02 16:08:14.349665 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:14.349624    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 02 16:08:18.458708 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:18.458677    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:21.706196 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.706159    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"]
Jun 02 16:08:21.706675 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.706481    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44" containerID="cri-o://f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c" gracePeriod=30
Jun 02 16:08:21.953310 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.953269    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"]
Jun 02 16:08:21.953629 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.953601    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container" containerID="cri-o://c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f" gracePeriod=30
Jun 02 16:08:21.953724 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.953655    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kube-rbac-proxy" containerID="cri-o://a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd" gracePeriod=30
Jun 02 16:08:21.979311 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.979278    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"]
Jun 02 16:08:21.983088 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.983068    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:21.987202 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.987181    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-411bf-predictor-serving-cert\""
Jun 02 16:08:21.987305 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.987182    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-411bf-kube-rbac-proxy-sar-config\""
Jun 02 16:08:21.998933 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:21.998903    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"]
Jun 02 16:08:22.015260 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.015230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.015430 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.015289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wv2x\" (UniqueName: \"kubernetes.io/projected/e71a98d0-c400-467d-97d1-04dfa6d078ee-kube-api-access-8wv2x\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.015430 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.015372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-411bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e71a98d0-c400-467d-97d1-04dfa6d078ee-error-404-isvc-411bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.115939 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.115907    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8wv2x\" (UniqueName: \"kubernetes.io/projected/e71a98d0-c400-467d-97d1-04dfa6d078ee-kube-api-access-8wv2x\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.116130 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.115965    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-411bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e71a98d0-c400-467d-97d1-04dfa6d078ee-error-404-isvc-411bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.116130 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.116009    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.116130 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:22.116102    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-411bf-predictor-serving-cert: secret "error-404-isvc-411bf-predictor-serving-cert" not found
Jun 02 16:08:22.116282 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:22.116164    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls podName:e71a98d0-c400-467d-97d1-04dfa6d078ee nodeName:}" failed. No retries permitted until 2026-06-02 16:08:22.616144935 +0000 UTC m=+1306.958566134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls") pod "error-404-isvc-411bf-predictor-75d76dc5cd-272s7" (UID: "e71a98d0-c400-467d-97d1-04dfa6d078ee") : secret "error-404-isvc-411bf-predictor-serving-cert" not found
Jun 02 16:08:22.116662 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.116641    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-411bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e71a98d0-c400-467d-97d1-04dfa6d078ee-error-404-isvc-411bf-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.124660 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.124631    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wv2x\" (UniqueName: \"kubernetes.io/projected/e71a98d0-c400-467d-97d1-04dfa6d078ee-kube-api-access-8wv2x\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.486629 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.486589    2577 generic.go:358] "Generic (PLEG): container finished" podID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerID="a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd" exitCode=2
Jun 02 16:08:22.486821 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.486662    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" event={"ID":"0990edda-a3df-4e13-ac79-c3601c8849aa","Type":"ContainerDied","Data":"a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd"}
Jun 02 16:08:22.620757 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.620719    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.623095 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.623074    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls\") pod \"error-404-isvc-411bf-predictor-75d76dc5cd-272s7\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") " pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:22.893483 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:22.893445    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:23.047694 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:23.047661    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"]
Jun 02 16:08:23.050939 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:08:23.050914    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71a98d0_c400_467d_97d1_04dfa6d078ee.slice/crio-a341510259053f1dcb3551e35f1d00dff7a86103fade352555e8d026a1afb803 WatchSource:0}: Error finding container a341510259053f1dcb3551e35f1d00dff7a86103fade352555e8d026a1afb803: Status 404 returned error can't find the container with id a341510259053f1dcb3551e35f1d00dff7a86103fade352555e8d026a1afb803
Jun 02 16:08:23.457794 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:23.457693    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:23.492578 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:23.492544    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" event={"ID":"e71a98d0-c400-467d-97d1-04dfa6d078ee","Type":"ContainerStarted","Data":"5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc"}
Jun 02 16:08:23.492578 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:23.492583    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" event={"ID":"e71a98d0-c400-467d-97d1-04dfa6d078ee","Type":"ContainerStarted","Data":"a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be"}
Jun 02 16:08:23.492800 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:23.492595    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" event={"ID":"e71a98d0-c400-467d-97d1-04dfa6d078ee","Type":"ContainerStarted","Data":"a341510259053f1dcb3551e35f1d00dff7a86103fade352555e8d026a1afb803"}
Jun 02 16:08:23.492800 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:23.492690    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:23.510037 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:23.509976    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podStartSLOduration=2.509953709 podStartE2EDuration="2.509953709s" podCreationTimestamp="2026-06-02 16:08:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:08:23.509424757 +0000 UTC m=+1307.851845984" watchObservedRunningTime="2026-06-02 16:08:23.509953709 +0000 UTC m=+1307.852374933"
Jun 02 16:08:24.350176 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:24.350146    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:08:24.499675 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:24.499643    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:24.500885 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:24.500849    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jun 02 16:08:25.250353 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.250309    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jun 02 16:08:25.389999 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.389972    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:08:25.444167 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.444085    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-5da44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0990edda-a3df-4e13-ac79-c3601c8849aa-error-404-isvc-5da44-kube-rbac-proxy-sar-config\") pod \"0990edda-a3df-4e13-ac79-c3601c8849aa\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") "
Jun 02 16:08:25.444167 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.444149    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2g6mx\" (UniqueName: \"kubernetes.io/projected/0990edda-a3df-4e13-ac79-c3601c8849aa-kube-api-access-2g6mx\") pod \"0990edda-a3df-4e13-ac79-c3601c8849aa\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") "
Jun 02 16:08:25.444343 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.444175    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0990edda-a3df-4e13-ac79-c3601c8849aa-proxy-tls\") pod \"0990edda-a3df-4e13-ac79-c3601c8849aa\" (UID: \"0990edda-a3df-4e13-ac79-c3601c8849aa\") "
Jun 02 16:08:25.444493 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.444470    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0990edda-a3df-4e13-ac79-c3601c8849aa-error-404-isvc-5da44-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-5da44-kube-rbac-proxy-sar-config") pod "0990edda-a3df-4e13-ac79-c3601c8849aa" (UID: "0990edda-a3df-4e13-ac79-c3601c8849aa"). InnerVolumeSpecName "error-404-isvc-5da44-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:08:25.446211 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.446182    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0990edda-a3df-4e13-ac79-c3601c8849aa-kube-api-access-2g6mx" (OuterVolumeSpecName: "kube-api-access-2g6mx") pod "0990edda-a3df-4e13-ac79-c3601c8849aa" (UID: "0990edda-a3df-4e13-ac79-c3601c8849aa"). InnerVolumeSpecName "kube-api-access-2g6mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:08:25.446211 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.446197    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0990edda-a3df-4e13-ac79-c3601c8849aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0990edda-a3df-4e13-ac79-c3601c8849aa" (UID: "0990edda-a3df-4e13-ac79-c3601c8849aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:08:25.501512 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.501477    2577 generic.go:358] "Generic (PLEG): container finished" podID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerID="c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f" exitCode=0
Jun 02 16:08:25.501680 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.501556    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"
Jun 02 16:08:25.501680 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.501567    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" event={"ID":"0990edda-a3df-4e13-ac79-c3601c8849aa","Type":"ContainerDied","Data":"c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f"}
Jun 02 16:08:25.501680 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.501604    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx" event={"ID":"0990edda-a3df-4e13-ac79-c3601c8849aa","Type":"ContainerDied","Data":"0c494004d027f1a700afcf26b2289ae7768946875573a0f19a0ab2f6227d4793"}
Jun 02 16:08:25.501680 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.501619    2577 scope.go:117] "RemoveContainer" containerID="a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd"
Jun 02 16:08:25.502206 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.502165    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jun 02 16:08:25.509907 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.509886    2577 scope.go:117] "RemoveContainer" containerID="c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f"
Jun 02 16:08:25.517433 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.517414    2577 scope.go:117] "RemoveContainer" containerID="a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd"
Jun 02 16:08:25.517702 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:25.517680    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd\": container with ID starting with a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd not found: ID does not exist" containerID="a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd"
Jun 02 16:08:25.517751 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.517710    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd"} err="failed to get container status \"a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd\": rpc error: code = NotFound desc = could not find container \"a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd\": container with ID starting with a5bebffe1172a3fc5e5bf2f18aa4e658b26acb7f291cad90aa8c44f490931cfd not found: ID does not exist"
Jun 02 16:08:25.517751 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.517729    2577 scope.go:117] "RemoveContainer" containerID="c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f"
Jun 02 16:08:25.517955 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:25.517935    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f\": container with ID starting with c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f not found: ID does not exist" containerID="c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f"
Jun 02 16:08:25.517995 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.517961    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f"} err="failed to get container status \"c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f\": rpc error: code = NotFound desc = could not find container \"c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f\": container with ID starting with c5456a692f64de2a1a1fc1b30f0ccd4d6a612b90a83491c6604ff38c2142c80f not found: ID does not exist"
Jun 02 16:08:25.523687 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.523661    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"]
Jun 02 16:08:25.531199 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.531173    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-5da44-predictor-98d5465cd-hvgvx"]
Jun 02 16:08:25.545666 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.545637    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0990edda-a3df-4e13-ac79-c3601c8849aa-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:08:25.545666 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.545664    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-5da44-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0990edda-a3df-4e13-ac79-c3601c8849aa-error-404-isvc-5da44-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:08:25.545783 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:25.545679    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g6mx\" (UniqueName: \"kubernetes.io/projected/0990edda-a3df-4e13-ac79-c3601c8849aa-kube-api-access-2g6mx\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:08:26.223002 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:26.222971    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" path="/var/lib/kubelet/pods/0990edda-a3df-4e13-ac79-c3601c8849aa/volumes"
Jun 02 16:08:28.457585 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:28.457544    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:30.506843 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:30.506816    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:08:30.507367 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:30.507342    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jun 02 16:08:33.456908 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:33.456864    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:33.457457 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:33.456996    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:35.815655 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.815621    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"]
Jun 02 16:08:35.816039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.815962    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kube-rbac-proxy"
Jun 02 16:08:35.816039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.815975    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kube-rbac-proxy"
Jun 02 16:08:35.816039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.815994    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container"
Jun 02 16:08:35.816039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.816001    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container"
Jun 02 16:08:35.816165 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.816058    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kube-rbac-proxy"
Jun 02 16:08:35.816165 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.816069    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0990edda-a3df-4e13-ac79-c3601c8849aa" containerName="kserve-container"
Jun 02 16:08:35.818844 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.818824    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:35.821110 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.821087    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-94f66-kube-rbac-proxy-sar-config\""
Jun 02 16:08:35.821243 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.821087    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-94f66-serving-cert\""
Jun 02 16:08:35.829923 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.829899    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"]
Jun 02 16:08:35.934985 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.934947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8466ec08-ef10-4121-9884-323dd26c7caf-openshift-service-ca-bundle\") pod \"ensemble-graph-94f66-f58d9dd86-bfvnv\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") " pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:35.935167 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:35.934995    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls\") pod \"ensemble-graph-94f66-f58d9dd86-bfvnv\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") " pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:36.036195 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:36.036156    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8466ec08-ef10-4121-9884-323dd26c7caf-openshift-service-ca-bundle\") pod \"ensemble-graph-94f66-f58d9dd86-bfvnv\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") " pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:36.036195 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:36.036199    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls\") pod \"ensemble-graph-94f66-f58d9dd86-bfvnv\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") " pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:36.036488 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:36.036297    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-94f66-serving-cert: secret "ensemble-graph-94f66-serving-cert" not found
Jun 02 16:08:36.036488 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:36.036364    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls podName:8466ec08-ef10-4121-9884-323dd26c7caf nodeName:}" failed. No retries permitted until 2026-06-02 16:08:36.536344984 +0000 UTC m=+1320.878766184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls") pod "ensemble-graph-94f66-f58d9dd86-bfvnv" (UID: "8466ec08-ef10-4121-9884-323dd26c7caf") : secret "ensemble-graph-94f66-serving-cert" not found
Jun 02 16:08:36.036912 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:36.036885    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8466ec08-ef10-4121-9884-323dd26c7caf-openshift-service-ca-bundle\") pod \"ensemble-graph-94f66-f58d9dd86-bfvnv\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") " pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:36.540810 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:36.540722    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls\") pod \"ensemble-graph-94f66-f58d9dd86-bfvnv\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") " pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:36.543024 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:36.542995    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls\") pod \"ensemble-graph-94f66-f58d9dd86-bfvnv\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") " pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:36.729670 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:36.729638    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:36.860229 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:36.860196    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"]
Jun 02 16:08:37.541070 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:37.541029    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" event={"ID":"8466ec08-ef10-4121-9884-323dd26c7caf","Type":"ContainerStarted","Data":"2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106"}
Jun 02 16:08:37.541070 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:37.541076    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" event={"ID":"8466ec08-ef10-4121-9884-323dd26c7caf","Type":"ContainerStarted","Data":"530269f30c4f577e89027c8946778807d53847c4c24049acfada0d812680e00b"}
Jun 02 16:08:37.541279 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:37.541157    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:37.560407 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:37.560331    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" podStartSLOduration=2.560316412 podStartE2EDuration="2.560316412s" podCreationTimestamp="2026-06-02 16:08:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:08:37.558373516 +0000 UTC m=+1321.900794742" watchObservedRunningTime="2026-06-02 16:08:37.560316412 +0000 UTC m=+1321.902737674"
Jun 02 16:08:38.456797 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:38.456762    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:40.507436 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:40.507384    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jun 02 16:08:43.457416 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:43.457342    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:43.549894 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:43.549859    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:08:48.456996 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:48.456954    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:08:50.508093 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:50.508048    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jun 02 16:08:51.766457 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:51.766425    2577 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cce28d2_504f_4153_975d_58978a4367c3.slice/crio-f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cce28d2_504f_4153_975d_58978a4367c3.slice/crio-conmon-f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c.scope\": RecentStats: unable to find data in memory cache]"
Jun 02 16:08:51.888621 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:51.888590    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:51.973529 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:51.973438    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cce28d2-504f-4153-975d-58978a4367c3-proxy-tls\") pod \"7cce28d2-504f-4153-975d-58978a4367c3\" (UID: \"7cce28d2-504f-4153-975d-58978a4367c3\") "
Jun 02 16:08:51.973529 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:51.973495    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cce28d2-504f-4153-975d-58978a4367c3-openshift-service-ca-bundle\") pod \"7cce28d2-504f-4153-975d-58978a4367c3\" (UID: \"7cce28d2-504f-4153-975d-58978a4367c3\") "
Jun 02 16:08:51.973892 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:51.973866    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cce28d2-504f-4153-975d-58978a4367c3-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7cce28d2-504f-4153-975d-58978a4367c3" (UID: "7cce28d2-504f-4153-975d-58978a4367c3"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:08:51.975592 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:51.975567    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cce28d2-504f-4153-975d-58978a4367c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7cce28d2-504f-4153-975d-58978a4367c3" (UID: "7cce28d2-504f-4153-975d-58978a4367c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:08:52.074553 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.074512    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7cce28d2-504f-4153-975d-58978a4367c3-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:08:52.074553 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.074546    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cce28d2-504f-4153-975d-58978a4367c3-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:08:52.591857 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.591771    2577 generic.go:358] "Generic (PLEG): container finished" podID="7cce28d2-504f-4153-975d-58978a4367c3" containerID="f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c" exitCode=137
Jun 02 16:08:52.591857 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.591833    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"
Jun 02 16:08:52.592031 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.591832    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" event={"ID":"7cce28d2-504f-4153-975d-58978a4367c3","Type":"ContainerDied","Data":"f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c"}
Jun 02 16:08:52.592031 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.591929    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp" event={"ID":"7cce28d2-504f-4153-975d-58978a4367c3","Type":"ContainerDied","Data":"f34d4b02f2141a1b684341c0fafc4bb1160eeba3263d27392151655d57318b74"}
Jun 02 16:08:52.592031 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.591948    2577 scope.go:117] "RemoveContainer" containerID="f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c"
Jun 02 16:08:52.600014 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.599983    2577 scope.go:117] "RemoveContainer" containerID="f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c"
Jun 02 16:08:52.600251 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:08:52.600234    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c\": container with ID starting with f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c not found: ID does not exist" containerID="f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c"
Jun 02 16:08:52.600316 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.600258    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c"} err="failed to get container status \"f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c\": rpc error: code = NotFound desc = could not find container \"f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c\": container with ID starting with f162a70ef916ac1cde2c7326ae80d06cd197b4a75f22460de972894a7e94df7c not found: ID does not exist"
Jun 02 16:08:52.609106 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.609082    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"]
Jun 02 16:08:52.613457 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:52.613437    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5da44-7795d6f48-6llxp"]
Jun 02 16:08:54.222497 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:08:54.222456    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cce28d2-504f-4153-975d-58978a4367c3" path="/var/lib/kubelet/pods/7cce28d2-504f-4153-975d-58978a4367c3/volumes"
Jun 02 16:09:00.507541 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:00.507459    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jun 02 16:09:10.508564 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:10.508532    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:09:21.993471 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:21.993427    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"]
Jun 02 16:09:21.993880 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:21.993867    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44"
Jun 02 16:09:21.993922 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:21.993882    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44"
Jun 02 16:09:21.993957 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:21.993948    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="7cce28d2-504f-4153-975d-58978a4367c3" containerName="sequence-graph-5da44"
Jun 02 16:09:21.996867 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:21.996851    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:21.999950 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:09:21.999917    2577 reflector.go:200] "Failed to watch" err="failed to list *v1.Secret: secrets \"sequence-graph-411bf-serving-cert\" is forbidden: User \"system:node:ip-10-0-133-168.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-133-168.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-411bf-serving-cert\"" type="*v1.Secret"
Jun 02 16:09:22.000329 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:22.000298    2577 status_manager.go:895] "Failed to get status for pod" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" err="pods \"sequence-graph-411bf-d968dbbf-78445\" is forbidden: User \"system:node:ip-10-0-133-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-133-168.ec2.internal' and this object"
Jun 02 16:09:22.000505 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:09:22.000480    2577 reflector.go:200] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"sequence-graph-411bf-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-133-168.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-133-168.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-411bf-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Jun 02 16:09:22.014464 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:22.014436    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"]
Jun 02 16:09:22.033774 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:22.033744    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-openshift-service-ca-bundle\") pod \"sequence-graph-411bf-d968dbbf-78445\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") " pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:22.033915 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:22.033787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls\") pod \"sequence-graph-411bf-d968dbbf-78445\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") " pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:22.134926 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:22.134891    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls\") pod \"sequence-graph-411bf-d968dbbf-78445\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") " pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:22.135147 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:22.135004    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-openshift-service-ca-bundle\") pod \"sequence-graph-411bf-d968dbbf-78445\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") " pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:22.135665 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:22.135642    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-openshift-service-ca-bundle\") pod \"sequence-graph-411bf-d968dbbf-78445\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") " pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:23.136123 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:09:23.136086    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-411bf-serving-cert: failed to sync secret cache: timed out waiting for the condition
Jun 02 16:09:23.136552 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:09:23.136169    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls podName:7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf nodeName:}" failed. No retries permitted until 2026-06-02 16:09:23.636151399 +0000 UTC m=+1367.978572598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls") pod "sequence-graph-411bf-d968dbbf-78445" (UID: "7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf") : failed to sync secret cache: timed out waiting for the condition
Jun 02 16:09:23.314519 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:23.314474    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-411bf-kube-rbac-proxy-sar-config\""
Jun 02 16:09:23.486893 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:23.486862    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-411bf-serving-cert\""
Jun 02 16:09:23.647145 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:23.647095    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls\") pod \"sequence-graph-411bf-d968dbbf-78445\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") " pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:23.649643 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:23.649615    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls\") pod \"sequence-graph-411bf-d968dbbf-78445\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") " pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:23.807779 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:23.807685    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:23.937027 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:23.937000    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"]
Jun 02 16:09:23.939604 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:09:23.939576    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f20e9_0a9b_4ae9_a989_045dcba3cdaf.slice/crio-b058949294316a6cde5019c57164724470e890918ce37c80aef207610a685cf8 WatchSource:0}: Error finding container b058949294316a6cde5019c57164724470e890918ce37c80aef207610a685cf8: Status 404 returned error can't find the container with id b058949294316a6cde5019c57164724470e890918ce37c80aef207610a685cf8
Jun 02 16:09:24.690276 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:24.690236    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" event={"ID":"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf","Type":"ContainerStarted","Data":"afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54"}
Jun 02 16:09:24.690276 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:24.690276    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" event={"ID":"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf","Type":"ContainerStarted","Data":"b058949294316a6cde5019c57164724470e890918ce37c80aef207610a685cf8"}
Jun 02 16:09:24.690775 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:24.690363    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:09:24.707718 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:24.707661    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" podStartSLOduration=3.707641379 podStartE2EDuration="3.707641379s" podCreationTimestamp="2026-06-02 16:09:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:09:24.70700312 +0000 UTC m=+1369.049424346" watchObservedRunningTime="2026-06-02 16:09:24.707641379 +0000 UTC m=+1369.050062600"
Jun 02 16:09:30.699070 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:09:30.699039    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:16:50.562490 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.562453    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"]
Jun 02 16:16:50.562924 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.562698    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66" containerID="cri-o://2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106" gracePeriod=30
Jun 02 16:16:50.792347 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.792314    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"]
Jun 02 16:16:50.792632 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.792611    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container" containerID="cri-o://f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a" gracePeriod=30
Jun 02 16:16:50.792697 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.792668    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kube-rbac-proxy" containerID="cri-o://6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558" gracePeriod=30
Jun 02 16:16:50.815960 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.815899    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"]
Jun 02 16:16:50.819190 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.819174    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:50.821145 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.821120    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-022f7-predictor-serving-cert\""
Jun 02 16:16:50.821288 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.821147    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-022f7-kube-rbac-proxy-sar-config\""
Jun 02 16:16:50.830416 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.830377    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"]
Jun 02 16:16:50.905583 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.905555    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:50.905708 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.905628    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-022f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d5560a-75f7-446b-a1cb-49388f257cef-error-404-isvc-022f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:50.905708 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:50.905650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm9w9\" (UniqueName: \"kubernetes.io/projected/55d5560a-75f7-446b-a1cb-49388f257cef-kube-api-access-rm9w9\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.006476 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.006440    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-022f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d5560a-75f7-446b-a1cb-49388f257cef-error-404-isvc-022f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.006659 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.006483    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rm9w9\" (UniqueName: \"kubernetes.io/projected/55d5560a-75f7-446b-a1cb-49388f257cef-kube-api-access-rm9w9\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.006659 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.006542    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.006792 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:16:51.006667    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-022f7-predictor-serving-cert: secret "error-404-isvc-022f7-predictor-serving-cert" not found
Jun 02 16:16:51.006792 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:16:51.006720    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls podName:55d5560a-75f7-446b-a1cb-49388f257cef nodeName:}" failed. No retries permitted until 2026-06-02 16:16:51.506702223 +0000 UTC m=+1815.849123440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls") pod "error-404-isvc-022f7-predictor-59674bc84-vx8dn" (UID: "55d5560a-75f7-446b-a1cb-49388f257cef") : secret "error-404-isvc-022f7-predictor-serving-cert" not found
Jun 02 16:16:51.007196 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.007171    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-022f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d5560a-75f7-446b-a1cb-49388f257cef-error-404-isvc-022f7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.017658 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.017634    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm9w9\" (UniqueName: \"kubernetes.io/projected/55d5560a-75f7-446b-a1cb-49388f257cef-kube-api-access-rm9w9\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.076864 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.076775    2577 generic.go:358] "Generic (PLEG): container finished" podID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerID="6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558" exitCode=2
Jun 02 16:16:51.076864 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.076830    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" event={"ID":"49dd9b8f-7879-42a9-9817-fee62e3d47fd","Type":"ContainerDied","Data":"6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558"}
Jun 02 16:16:51.509928 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.509894    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.512440 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.512415    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls\") pod \"error-404-isvc-022f7-predictor-59674bc84-vx8dn\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") " pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.729826 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.729787    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:51.860382 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.860351    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"]
Jun 02 16:16:51.862761 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:16:51.862732    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d5560a_75f7_446b_a1cb_49388f257cef.slice/crio-77343d8e7cf26b02f5a6315c9b8b284e102fe4a4813a1cb0edcaad64d71acaf1 WatchSource:0}: Error finding container 77343d8e7cf26b02f5a6315c9b8b284e102fe4a4813a1cb0edcaad64d71acaf1: Status 404 returned error can't find the container with id 77343d8e7cf26b02f5a6315c9b8b284e102fe4a4813a1cb0edcaad64d71acaf1
Jun 02 16:16:51.864961 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:51.864943    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:16:52.081864 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:52.081762    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" event={"ID":"55d5560a-75f7-446b-a1cb-49388f257cef","Type":"ContainerStarted","Data":"42c70e2939f969989755196801c013d0b7b6027e5a61f915fdc62d159d0561cd"}
Jun 02 16:16:52.081864 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:52.081817    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" event={"ID":"55d5560a-75f7-446b-a1cb-49388f257cef","Type":"ContainerStarted","Data":"50b5dfc8f61a39f9e9215a62cc5f7903830661ef0ce66f75f8e3b7e649faa271"}
Jun 02 16:16:52.081864 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:52.081833    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" event={"ID":"55d5560a-75f7-446b-a1cb-49388f257cef","Type":"ContainerStarted","Data":"77343d8e7cf26b02f5a6315c9b8b284e102fe4a4813a1cb0edcaad64d71acaf1"}
Jun 02 16:16:52.082143 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:52.082021    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:52.082206 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:52.082195    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:52.083694 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:52.083667    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:16:52.100688 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:52.100638    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podStartSLOduration=2.100623796 podStartE2EDuration="2.100623796s" podCreationTimestamp="2026-06-02 16:16:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:16:52.098113233 +0000 UTC m=+1816.440534455" watchObservedRunningTime="2026-06-02 16:16:52.100623796 +0000 UTC m=+1816.443045017"
Jun 02 16:16:53.085843 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:53.085797    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:16:53.548820 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:53.548781    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:16:54.032664 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.032637    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:16:54.090303 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.090265    2577 generic.go:358] "Generic (PLEG): container finished" podID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerID="f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a" exitCode=0
Jun 02 16:16:54.090731 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.090339    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"
Jun 02 16:16:54.090731 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.090350    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" event={"ID":"49dd9b8f-7879-42a9-9817-fee62e3d47fd","Type":"ContainerDied","Data":"f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a"}
Jun 02 16:16:54.090731 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.090406    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2" event={"ID":"49dd9b8f-7879-42a9-9817-fee62e3d47fd","Type":"ContainerDied","Data":"012abfa9fd1fc137693a4f4d8ba1c52ef214eaebf89b994c50464363f2eee443"}
Jun 02 16:16:54.090731 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.090424    2577 scope.go:117] "RemoveContainer" containerID="6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558"
Jun 02 16:16:54.098061 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.098018    2577 scope.go:117] "RemoveContainer" containerID="f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a"
Jun 02 16:16:54.104987 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.104970    2577 scope.go:117] "RemoveContainer" containerID="6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558"
Jun 02 16:16:54.105244 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:16:54.105225    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558\": container with ID starting with 6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558 not found: ID does not exist" containerID="6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558"
Jun 02 16:16:54.105317 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.105254    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558"} err="failed to get container status \"6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558\": rpc error: code = NotFound desc = could not find container \"6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558\": container with ID starting with 6eb6d81654f244a42f03eb216cb9a3675facadfa0094281c08fcbf2c48773558 not found: ID does not exist"
Jun 02 16:16:54.105317 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.105272    2577 scope.go:117] "RemoveContainer" containerID="f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a"
Jun 02 16:16:54.105536 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:16:54.105519    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a\": container with ID starting with f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a not found: ID does not exist" containerID="f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a"
Jun 02 16:16:54.105578 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.105542    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a"} err="failed to get container status \"f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a\": rpc error: code = NotFound desc = could not find container \"f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a\": container with ID starting with f0fbf416d1683ca8217c7bb49350451a9d30eac280d9431dfe3f4fcaa47dc54a not found: ID does not exist"
Jun 02 16:16:54.130473 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.130445    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-94f66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49dd9b8f-7879-42a9-9817-fee62e3d47fd-error-404-isvc-94f66-kube-rbac-proxy-sar-config\") pod \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") "
Jun 02 16:16:54.130604 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.130497    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls\") pod \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") "
Jun 02 16:16:54.130604 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.130559    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9vq8\" (UniqueName: \"kubernetes.io/projected/49dd9b8f-7879-42a9-9817-fee62e3d47fd-kube-api-access-q9vq8\") pod \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\" (UID: \"49dd9b8f-7879-42a9-9817-fee62e3d47fd\") "
Jun 02 16:16:54.130839 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.130808    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49dd9b8f-7879-42a9-9817-fee62e3d47fd-error-404-isvc-94f66-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-94f66-kube-rbac-proxy-sar-config") pod "49dd9b8f-7879-42a9-9817-fee62e3d47fd" (UID: "49dd9b8f-7879-42a9-9817-fee62e3d47fd"). InnerVolumeSpecName "error-404-isvc-94f66-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:16:54.132526 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.132504    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49dd9b8f-7879-42a9-9817-fee62e3d47fd-kube-api-access-q9vq8" (OuterVolumeSpecName: "kube-api-access-q9vq8") pod "49dd9b8f-7879-42a9-9817-fee62e3d47fd" (UID: "49dd9b8f-7879-42a9-9817-fee62e3d47fd"). InnerVolumeSpecName "kube-api-access-q9vq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:16:54.132624 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.132543    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49dd9b8f-7879-42a9-9817-fee62e3d47fd" (UID: "49dd9b8f-7879-42a9-9817-fee62e3d47fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:16:54.231979 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.231954    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49dd9b8f-7879-42a9-9817-fee62e3d47fd-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:16:54.231979 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.231980    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vq8\" (UniqueName: \"kubernetes.io/projected/49dd9b8f-7879-42a9-9817-fee62e3d47fd-kube-api-access-q9vq8\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:16:54.232117 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.231991    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-94f66-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49dd9b8f-7879-42a9-9817-fee62e3d47fd-error-404-isvc-94f66-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:16:54.410432 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.410384    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"]
Jun 02 16:16:54.413772 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:54.413748    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-94f66-predictor-76446b9648-x8kt2"]
Jun 02 16:16:56.222470 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:56.222440    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" path="/var/lib/kubelet/pods/49dd9b8f-7879-42a9-9817-fee62e3d47fd/volumes"
Jun 02 16:16:58.090207 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:58.090178    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:16:58.090740 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:58.090714    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:16:58.548851 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:16:58.548809    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:03.548190 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:03.548147    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:03.548720 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:03.548288    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:17:08.090717 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:08.090674    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:17:08.548557 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:08.548520    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:13.548331 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:13.548290    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:18.090942 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:18.090903    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:17:18.548722 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:18.548681    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:20.706398 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:20.706367    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:17:20.760875 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:20.760836    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8466ec08-ef10-4121-9884-323dd26c7caf-openshift-service-ca-bundle\") pod \"8466ec08-ef10-4121-9884-323dd26c7caf\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") "
Jun 02 16:17:20.761076 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:20.760918    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls\") pod \"8466ec08-ef10-4121-9884-323dd26c7caf\" (UID: \"8466ec08-ef10-4121-9884-323dd26c7caf\") "
Jun 02 16:17:20.761232 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:20.761205    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8466ec08-ef10-4121-9884-323dd26c7caf-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8466ec08-ef10-4121-9884-323dd26c7caf" (UID: "8466ec08-ef10-4121-9884-323dd26c7caf"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:17:20.762996 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:20.762970    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8466ec08-ef10-4121-9884-323dd26c7caf" (UID: "8466ec08-ef10-4121-9884-323dd26c7caf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:17:20.862688 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:20.862590    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8466ec08-ef10-4121-9884-323dd26c7caf-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:17:20.862688 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:20.862629    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8466ec08-ef10-4121-9884-323dd26c7caf-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:17:21.172663 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.172627    2577 generic.go:358] "Generic (PLEG): container finished" podID="8466ec08-ef10-4121-9884-323dd26c7caf" containerID="2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106" exitCode=0
Jun 02 16:17:21.172838 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.172694    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"
Jun 02 16:17:21.172838 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.172710    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" event={"ID":"8466ec08-ef10-4121-9884-323dd26c7caf","Type":"ContainerDied","Data":"2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106"}
Jun 02 16:17:21.172838 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.172747    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv" event={"ID":"8466ec08-ef10-4121-9884-323dd26c7caf","Type":"ContainerDied","Data":"530269f30c4f577e89027c8946778807d53847c4c24049acfada0d812680e00b"}
Jun 02 16:17:21.172838 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.172769    2577 scope.go:117] "RemoveContainer" containerID="2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106"
Jun 02 16:17:21.180998 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.180971    2577 scope.go:117] "RemoveContainer" containerID="2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106"
Jun 02 16:17:21.181289 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:17:21.181267    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106\": container with ID starting with 2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106 not found: ID does not exist" containerID="2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106"
Jun 02 16:17:21.181377 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.181296    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106"} err="failed to get container status \"2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106\": rpc error: code = NotFound desc = could not find container \"2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106\": container with ID starting with 2123561f16205a11a6bd65ee8c131eca7ddb4e5eb2a8cdc0f98d203fb451b106 not found: ID does not exist"
Jun 02 16:17:21.193548 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.193521    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"]
Jun 02 16:17:21.198039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:21.198013    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-94f66-f58d9dd86-bfvnv"]
Jun 02 16:17:22.222236 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:22.222202    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" path="/var/lib/kubelet/pods/8466ec08-ef10-4121-9884-323dd26c7caf/volumes"
Jun 02 16:17:28.091244 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:28.091200    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 02 16:17:36.623144 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.623047    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"]
Jun 02 16:17:36.623540 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.623302    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf" containerID="cri-o://afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54" gracePeriod=30
Jun 02 16:17:36.818274 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.818241    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"]
Jun 02 16:17:36.818583 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.818556    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container" containerID="cri-o://a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be" gracePeriod=30
Jun 02 16:17:36.818673 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.818601    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kube-rbac-proxy" containerID="cri-o://5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc" gracePeriod=30
Jun 02 16:17:36.878671 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.878588    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"]
Jun 02 16:17:36.878984 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.878971    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kube-rbac-proxy"
Jun 02 16:17:36.879024 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.878987    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kube-rbac-proxy"
Jun 02 16:17:36.879024 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.879012    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container"
Jun 02 16:17:36.879024 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.879018    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container"
Jun 02 16:17:36.879121 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.879034    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66"
Jun 02 16:17:36.879121 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.879040    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66"
Jun 02 16:17:36.879121 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.879097    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="8466ec08-ef10-4121-9884-323dd26c7caf" containerName="ensemble-graph-94f66"
Jun 02 16:17:36.879121 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.879106    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kube-rbac-proxy"
Jun 02 16:17:36.879121 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.879113    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="49dd9b8f-7879-42a9-9817-fee62e3d47fd" containerName="kserve-container"
Jun 02 16:17:36.883572 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.883554    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:36.885248 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.885228    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78d4f-predictor-serving-cert\""
Jun 02 16:17:36.885346 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.885283    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-78d4f-kube-rbac-proxy-sar-config\""
Jun 02 16:17:36.892249 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.892223    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"]
Jun 02 16:17:36.999746 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.999701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-78d4f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-error-404-isvc-78d4f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:36.999944 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.999767    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-proxy-tls\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:36.999944 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:36.999827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp7lr\" (UniqueName: \"kubernetes.io/projected/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-kube-api-access-vp7lr\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:37.100949 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.100909    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-78d4f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-error-404-isvc-78d4f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:37.101178 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.100958    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-proxy-tls\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:37.101178 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.100980    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vp7lr\" (UniqueName: \"kubernetes.io/projected/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-kube-api-access-vp7lr\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:37.101660 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.101628    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-78d4f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-error-404-isvc-78d4f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:37.103356 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.103334    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-proxy-tls\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:37.110104 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.110083    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp7lr\" (UniqueName: \"kubernetes.io/projected/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-kube-api-access-vp7lr\") pod \"error-404-isvc-78d4f-predictor-79467495b8-z9tfb\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") " pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:37.194812 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.194770    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:37.224573 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.224534    2577 generic.go:358] "Generic (PLEG): container finished" podID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerID="5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc" exitCode=2
Jun 02 16:17:37.224726 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.224598    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" event={"ID":"e71a98d0-c400-467d-97d1-04dfa6d078ee","Type":"ContainerDied","Data":"5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc"}
Jun 02 16:17:37.326276 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:37.326243    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"]
Jun 02 16:17:37.329180 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:17:37.329153    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf88d9e_57a2_42ce_b89b_f63ef0df9719.slice/crio-f94a92e71e3285c1e7b796551b205098f6d81f1bc80c81c95e49545f574a6090 WatchSource:0}: Error finding container f94a92e71e3285c1e7b796551b205098f6d81f1bc80c81c95e49545f574a6090: Status 404 returned error can't find the container with id f94a92e71e3285c1e7b796551b205098f6d81f1bc80c81c95e49545f574a6090
Jun 02 16:17:38.091760 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:38.091734    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:17:38.228761 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:38.228724    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" event={"ID":"5cf88d9e-57a2-42ce-b89b-f63ef0df9719","Type":"ContainerStarted","Data":"ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5"}
Jun 02 16:17:38.228761 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:38.228762    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" event={"ID":"5cf88d9e-57a2-42ce-b89b-f63ef0df9719","Type":"ContainerStarted","Data":"bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98"}
Jun 02 16:17:38.228961 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:38.228773    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" event={"ID":"5cf88d9e-57a2-42ce-b89b-f63ef0df9719","Type":"ContainerStarted","Data":"f94a92e71e3285c1e7b796551b205098f6d81f1bc80c81c95e49545f574a6090"}
Jun 02 16:17:38.228961 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:38.228825    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:38.250074 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:38.250027    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podStartSLOduration=2.250012593 podStartE2EDuration="2.250012593s" podCreationTimestamp="2026-06-02 16:17:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:17:38.248526058 +0000 UTC m=+1862.590947291" watchObservedRunningTime="2026-06-02 16:17:38.250012593 +0000 UTC m=+1862.592433793"
Jun 02 16:17:39.232345 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:39.232311    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:39.233627 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:39.233598    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 02 16:17:40.066529 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.066509    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:17:40.226701 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.226660    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls\") pod \"e71a98d0-c400-467d-97d1-04dfa6d078ee\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") "
Jun 02 16:17:40.226886 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.226714    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-411bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e71a98d0-c400-467d-97d1-04dfa6d078ee-error-404-isvc-411bf-kube-rbac-proxy-sar-config\") pod \"e71a98d0-c400-467d-97d1-04dfa6d078ee\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") "
Jun 02 16:17:40.226886 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.226744    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wv2x\" (UniqueName: \"kubernetes.io/projected/e71a98d0-c400-467d-97d1-04dfa6d078ee-kube-api-access-8wv2x\") pod \"e71a98d0-c400-467d-97d1-04dfa6d078ee\" (UID: \"e71a98d0-c400-467d-97d1-04dfa6d078ee\") "
Jun 02 16:17:40.227108 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.227079    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71a98d0-c400-467d-97d1-04dfa6d078ee-error-404-isvc-411bf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-411bf-kube-rbac-proxy-sar-config") pod "e71a98d0-c400-467d-97d1-04dfa6d078ee" (UID: "e71a98d0-c400-467d-97d1-04dfa6d078ee"). InnerVolumeSpecName "error-404-isvc-411bf-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:17:40.228835 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.228810    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71a98d0-c400-467d-97d1-04dfa6d078ee-kube-api-access-8wv2x" (OuterVolumeSpecName: "kube-api-access-8wv2x") pod "e71a98d0-c400-467d-97d1-04dfa6d078ee" (UID: "e71a98d0-c400-467d-97d1-04dfa6d078ee"). InnerVolumeSpecName "kube-api-access-8wv2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:17:40.228942 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.228838    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e71a98d0-c400-467d-97d1-04dfa6d078ee" (UID: "e71a98d0-c400-467d-97d1-04dfa6d078ee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:17:40.236784 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.236756    2577 generic.go:358] "Generic (PLEG): container finished" podID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerID="a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be" exitCode=0
Jun 02 16:17:40.237113 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.236835    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" event={"ID":"e71a98d0-c400-467d-97d1-04dfa6d078ee","Type":"ContainerDied","Data":"a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be"}
Jun 02 16:17:40.237113 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.236850    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"
Jun 02 16:17:40.237113 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.236872    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7" event={"ID":"e71a98d0-c400-467d-97d1-04dfa6d078ee","Type":"ContainerDied","Data":"a341510259053f1dcb3551e35f1d00dff7a86103fade352555e8d026a1afb803"}
Jun 02 16:17:40.237113 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.236891    2577 scope.go:117] "RemoveContainer" containerID="5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc"
Jun 02 16:17:40.237607 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.237550    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 02 16:17:40.247763 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.247745    2577 scope.go:117] "RemoveContainer" containerID="a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be"
Jun 02 16:17:40.255000 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.254981    2577 scope.go:117] "RemoveContainer" containerID="5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc"
Jun 02 16:17:40.255257 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:17:40.255233    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc\": container with ID starting with 5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc not found: ID does not exist" containerID="5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc"
Jun 02 16:17:40.255334 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.255267    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc"} err="failed to get container status \"5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc\": rpc error: code = NotFound desc = could not find container \"5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc\": container with ID starting with 5519d573412aff96321810869b5a31937f14aca9efb575a27e072a17c8545cdc not found: ID does not exist"
Jun 02 16:17:40.255334 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.255285    2577 scope.go:117] "RemoveContainer" containerID="a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be"
Jun 02 16:17:40.255642 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:17:40.255619    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be\": container with ID starting with a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be not found: ID does not exist" containerID="a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be"
Jun 02 16:17:40.255711 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.255648    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be"} err="failed to get container status \"a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be\": rpc error: code = NotFound desc = could not find container \"a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be\": container with ID starting with a1509f8a68dafea2b6da0c0177c2edf11f7e58b28d6217539b39f1e0bca791be not found: ID does not exist"
Jun 02 16:17:40.261124 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.261103    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"]
Jun 02 16:17:40.263675 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.263654    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-411bf-predictor-75d76dc5cd-272s7"]
Jun 02 16:17:40.328184 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.328147    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e71a98d0-c400-467d-97d1-04dfa6d078ee-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:17:40.328184 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.328181    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-411bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e71a98d0-c400-467d-97d1-04dfa6d078ee-error-404-isvc-411bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:17:40.328468 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.328197    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wv2x\" (UniqueName: \"kubernetes.io/projected/e71a98d0-c400-467d-97d1-04dfa6d078ee-kube-api-access-8wv2x\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:17:40.698377 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:40.698341    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:42.222039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:42.222010    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" path="/var/lib/kubelet/pods/e71a98d0-c400-467d-97d1-04dfa6d078ee/volumes"
Jun 02 16:17:45.242170 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:45.242135    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:17:45.242747 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:45.242722    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 02 16:17:45.697767 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:45.697730    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:50.698172 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.698131    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:50.698654 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.698285    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:17:50.830336 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.830302    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"]
Jun 02 16:17:50.830706 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.830684    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kube-rbac-proxy"
Jun 02 16:17:50.830706 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.830701    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kube-rbac-proxy"
Jun 02 16:17:50.830804 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.830712    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container"
Jun 02 16:17:50.830804 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.830719    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container"
Jun 02 16:17:50.830804 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.830778    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kserve-container"
Jun 02 16:17:50.830804 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.830792    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="e71a98d0-c400-467d-97d1-04dfa6d078ee" containerName="kube-rbac-proxy"
Jun 02 16:17:50.835661 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.835644    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:50.837822 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.837796    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-022f7-serving-cert\""
Jun 02 16:17:50.837942 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.837796    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-022f7-kube-rbac-proxy-sar-config\""
Jun 02 16:17:50.842542 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.842515    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"]
Jun 02 16:17:50.912349 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.912310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-proxy-tls\") pod \"splitter-graph-022f7-65bb66f8cc-9hzd2\" (UID: \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\") " pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:50.912554 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:50.912378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-openshift-service-ca-bundle\") pod \"splitter-graph-022f7-65bb66f8cc-9hzd2\" (UID: \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\") " pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:51.013615 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:51.013537    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-proxy-tls\") pod \"splitter-graph-022f7-65bb66f8cc-9hzd2\" (UID: \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\") " pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:51.013615 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:51.013594    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-openshift-service-ca-bundle\") pod \"splitter-graph-022f7-65bb66f8cc-9hzd2\" (UID: \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\") " pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:51.014151 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:51.014121    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-openshift-service-ca-bundle\") pod \"splitter-graph-022f7-65bb66f8cc-9hzd2\" (UID: \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\") " pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:51.015963 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:51.015939    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-proxy-tls\") pod \"splitter-graph-022f7-65bb66f8cc-9hzd2\" (UID: \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\") " pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:51.146935 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:51.146883    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:51.274218 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:51.274108    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"]
Jun 02 16:17:51.276745 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:17:51.276716    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0572d6e5_fdf0_4d96_a572_cc8fb5d51548.slice/crio-3ef72f126fd7f3afb5cb164e7e44cb09b0bec23ef0ee3f5369814877aaa1f0a9 WatchSource:0}: Error finding container 3ef72f126fd7f3afb5cb164e7e44cb09b0bec23ef0ee3f5369814877aaa1f0a9: Status 404 returned error can't find the container with id 3ef72f126fd7f3afb5cb164e7e44cb09b0bec23ef0ee3f5369814877aaa1f0a9
Jun 02 16:17:52.273751 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:52.273711    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" event={"ID":"0572d6e5-fdf0-4d96-a572-cc8fb5d51548","Type":"ContainerStarted","Data":"472ee33f47e27403c1c1ea46a95c72ead9207edafdded2133a4997e7278bc96b"}
Jun 02 16:17:52.273751 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:52.273754    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" event={"ID":"0572d6e5-fdf0-4d96-a572-cc8fb5d51548","Type":"ContainerStarted","Data":"3ef72f126fd7f3afb5cb164e7e44cb09b0bec23ef0ee3f5369814877aaa1f0a9"}
Jun 02 16:17:52.274208 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:52.273803    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:17:52.291041 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:52.290991    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" podStartSLOduration=2.290978023 podStartE2EDuration="2.290978023s" podCreationTimestamp="2026-06-02 16:17:50 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:17:52.289691619 +0000 UTC m=+1876.632112853" watchObservedRunningTime="2026-06-02 16:17:52.290978023 +0000 UTC m=+1876.633399247"
Jun 02 16:17:55.242708 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:55.242626    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 02 16:17:55.698015 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:55.697977    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:17:58.282008 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:17:58.281976    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:18:00.697340 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:00.697301    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:18:00.891460 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:00.891357    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"]
Jun 02 16:18:00.891606 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:00.891581    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7" containerID="cri-o://472ee33f47e27403c1c1ea46a95c72ead9207edafdded2133a4997e7278bc96b" gracePeriod=30
Jun 02 16:18:00.971725 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:00.971690    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"]
Jun 02 16:18:00.972000 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:00.971970    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container" containerID="cri-o://50b5dfc8f61a39f9e9215a62cc5f7903830661ef0ce66f75f8e3b7e649faa271" gracePeriod=30
Jun 02 16:18:00.972133 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:00.972004    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kube-rbac-proxy" containerID="cri-o://42c70e2939f969989755196801c013d0b7b6027e5a61f915fdc62d159d0561cd" gracePeriod=30
Jun 02 16:18:01.172276 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.172244    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"]
Jun 02 16:18:01.175700 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.175682    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.177573 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.177553    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cb302-kube-rbac-proxy-sar-config\""
Jun 02 16:18:01.177672 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.177553    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cb302-predictor-serving-cert\""
Jun 02 16:18:01.185517 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.185377    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"]
Jun 02 16:18:01.197739 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.197712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-proxy-tls\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.197908 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.197764    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"error-404-isvc-cb302-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-error-404-isvc-cb302-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.197908 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.197818    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr652\" (UniqueName: \"kubernetes.io/projected/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-kube-api-access-xr652\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.298711 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.298682    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-proxy-tls\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.298883 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.298736    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"error-404-isvc-cb302-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-error-404-isvc-cb302-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.298883 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.298767    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xr652\" (UniqueName: \"kubernetes.io/projected/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-kube-api-access-xr652\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.299690 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.299659    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"error-404-isvc-cb302-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-error-404-isvc-cb302-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.301410 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.301372    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-proxy-tls\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.303053 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.303031    2577 generic.go:358] "Generic (PLEG): container finished" podID="55d5560a-75f7-446b-a1cb-49388f257cef" containerID="42c70e2939f969989755196801c013d0b7b6027e5a61f915fdc62d159d0561cd" exitCode=2
Jun 02 16:18:01.303126 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.303094    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" event={"ID":"55d5560a-75f7-446b-a1cb-49388f257cef","Type":"ContainerDied","Data":"42c70e2939f969989755196801c013d0b7b6027e5a61f915fdc62d159d0561cd"}
Jun 02 16:18:01.308771 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.308753    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr652\" (UniqueName: \"kubernetes.io/projected/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-kube-api-access-xr652\") pod \"error-404-isvc-cb302-predictor-6f58d6db6-sp4rg\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") " pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.487479 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.487363    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:01.620496 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:01.620456    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"]
Jun 02 16:18:01.623808 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:18:01.623772    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c59584_eaab_4fd0_b6af_fb75dd0abef1.slice/crio-f42df99fedd04401b757a2963e459399b51cd897967a14c8aa0b22af4c688913 WatchSource:0}: Error finding container f42df99fedd04401b757a2963e459399b51cd897967a14c8aa0b22af4c688913: Status 404 returned error can't find the container with id f42df99fedd04401b757a2963e459399b51cd897967a14c8aa0b22af4c688913
Jun 02 16:18:02.308217 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:02.308180    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" event={"ID":"b0c59584-eaab-4fd0-b6af-fb75dd0abef1","Type":"ContainerStarted","Data":"5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb"}
Jun 02 16:18:02.308217 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:02.308221    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" event={"ID":"b0c59584-eaab-4fd0-b6af-fb75dd0abef1","Type":"ContainerStarted","Data":"4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13"}
Jun 02 16:18:02.308710 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:02.308234    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" event={"ID":"b0c59584-eaab-4fd0-b6af-fb75dd0abef1","Type":"ContainerStarted","Data":"f42df99fedd04401b757a2963e459399b51cd897967a14c8aa0b22af4c688913"}
Jun 02 16:18:02.308710 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:02.308351    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:02.328267 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:02.328216    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podStartSLOduration=1.3281979 podStartE2EDuration="1.3281979s" podCreationTimestamp="2026-06-02 16:18:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:18:02.32602388 +0000 UTC m=+1886.668445112" watchObservedRunningTime="2026-06-02 16:18:02.3281979 +0000 UTC m=+1886.670619123"
Jun 02 16:18:03.086702 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:03.086517    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jun 02 16:18:03.281633 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:03.281587    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:18:03.312234 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:03.312198    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:03.313070 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:03.313047    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 02 16:18:04.317594 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.317554    2577 generic.go:358] "Generic (PLEG): container finished" podID="55d5560a-75f7-446b-a1cb-49388f257cef" containerID="50b5dfc8f61a39f9e9215a62cc5f7903830661ef0ce66f75f8e3b7e649faa271" exitCode=0
Jun 02 16:18:04.318032 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.317629    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" event={"ID":"55d5560a-75f7-446b-a1cb-49388f257cef","Type":"ContainerDied","Data":"50b5dfc8f61a39f9e9215a62cc5f7903830661ef0ce66f75f8e3b7e649faa271"}
Jun 02 16:18:04.318096 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.318038    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 02 16:18:04.437782 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.437724    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:18:04.528999 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.528962    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rm9w9\" (UniqueName: \"kubernetes.io/projected/55d5560a-75f7-446b-a1cb-49388f257cef-kube-api-access-rm9w9\") pod \"55d5560a-75f7-446b-a1cb-49388f257cef\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") "
Jun 02 16:18:04.529193 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.529022    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls\") pod \"55d5560a-75f7-446b-a1cb-49388f257cef\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") "
Jun 02 16:18:04.529193 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.529077    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-022f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d5560a-75f7-446b-a1cb-49388f257cef-error-404-isvc-022f7-kube-rbac-proxy-sar-config\") pod \"55d5560a-75f7-446b-a1cb-49388f257cef\" (UID: \"55d5560a-75f7-446b-a1cb-49388f257cef\") "
Jun 02 16:18:04.529435 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.529402    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d5560a-75f7-446b-a1cb-49388f257cef-error-404-isvc-022f7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-022f7-kube-rbac-proxy-sar-config") pod "55d5560a-75f7-446b-a1cb-49388f257cef" (UID: "55d5560a-75f7-446b-a1cb-49388f257cef"). InnerVolumeSpecName "error-404-isvc-022f7-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:18:04.531091 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.531071    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "55d5560a-75f7-446b-a1cb-49388f257cef" (UID: "55d5560a-75f7-446b-a1cb-49388f257cef"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:18:04.531199 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.531181    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d5560a-75f7-446b-a1cb-49388f257cef-kube-api-access-rm9w9" (OuterVolumeSpecName: "kube-api-access-rm9w9") pod "55d5560a-75f7-446b-a1cb-49388f257cef" (UID: "55d5560a-75f7-446b-a1cb-49388f257cef"). InnerVolumeSpecName "kube-api-access-rm9w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:18:04.629821 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.629785    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm9w9\" (UniqueName: \"kubernetes.io/projected/55d5560a-75f7-446b-a1cb-49388f257cef-kube-api-access-rm9w9\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:18:04.629821 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.629822    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/55d5560a-75f7-446b-a1cb-49388f257cef-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:18:04.630035 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:04.629838    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-022f7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/55d5560a-75f7-446b-a1cb-49388f257cef-error-404-isvc-022f7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:18:05.242681 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:05.242645    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 02 16:18:05.321993 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:05.321956    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn" event={"ID":"55d5560a-75f7-446b-a1cb-49388f257cef","Type":"ContainerDied","Data":"77343d8e7cf26b02f5a6315c9b8b284e102fe4a4813a1cb0edcaad64d71acaf1"}
Jun 02 16:18:05.322448 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:05.322000    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"
Jun 02 16:18:05.322448 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:05.322009    2577 scope.go:117] "RemoveContainer" containerID="42c70e2939f969989755196801c013d0b7b6027e5a61f915fdc62d159d0561cd"
Jun 02 16:18:05.330993 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:05.330975    2577 scope.go:117] "RemoveContainer" containerID="50b5dfc8f61a39f9e9215a62cc5f7903830661ef0ce66f75f8e3b7e649faa271"
Jun 02 16:18:05.345176 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:05.345150    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"]
Jun 02 16:18:05.348884 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:05.348860    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-022f7-predictor-59674bc84-vx8dn"]
Jun 02 16:18:05.698071 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:05.698030    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:18:06.222954 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:06.222920    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" path="/var/lib/kubelet/pods/55d5560a-75f7-446b-a1cb-49388f257cef/volumes"
Jun 02 16:18:06.772117 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:06.772095    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:18:06.847559 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:06.847526    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls\") pod \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") "
Jun 02 16:18:06.847732 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:06.847567    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-openshift-service-ca-bundle\") pod \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\" (UID: \"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf\") "
Jun 02 16:18:06.847956 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:06.847933    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" (UID: "7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:18:06.849617 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:06.849594    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" (UID: "7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:18:06.948335 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:06.948302    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:18:06.948335 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:06.948333    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:18:07.331547 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.331464    2577 generic.go:358] "Generic (PLEG): container finished" podID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerID="afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54" exitCode=0
Jun 02 16:18:07.331547 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.331516    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" event={"ID":"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf","Type":"ContainerDied","Data":"afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54"}
Jun 02 16:18:07.331547 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.331523    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"
Jun 02 16:18:07.331547 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.331542    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445" event={"ID":"7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf","Type":"ContainerDied","Data":"b058949294316a6cde5019c57164724470e890918ce37c80aef207610a685cf8"}
Jun 02 16:18:07.331808 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.331558    2577 scope.go:117] "RemoveContainer" containerID="afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54"
Jun 02 16:18:07.339914 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.339896    2577 scope.go:117] "RemoveContainer" containerID="afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54"
Jun 02 16:18:07.340161 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:18:07.340144    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54\": container with ID starting with afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54 not found: ID does not exist" containerID="afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54"
Jun 02 16:18:07.340209 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.340168    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54"} err="failed to get container status \"afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54\": rpc error: code = NotFound desc = could not find container \"afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54\": container with ID starting with afa5e98973d2217595b5437808dff9ac49a855a007026c910cb4c18f0a196a54 not found: ID does not exist"
Jun 02 16:18:07.352502 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.352474    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"]
Jun 02 16:18:07.358097 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:07.358077    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-411bf-d968dbbf-78445"]
Jun 02 16:18:08.227207 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:08.227167    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" path="/var/lib/kubelet/pods/7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf/volumes"
Jun 02 16:18:08.281157 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:08.281108    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:18:09.322111 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:09.322082    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:18:09.322608 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:09.322581    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 02 16:18:13.281267 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:13.281230    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:18:13.281681 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:13.281335    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:18:15.242920 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:15.242886    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 02 16:18:18.281346 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:18.281291    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:18:19.323082 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:19.323045    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 02 16:18:23.280929 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:23.280883    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:18:25.243575 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:25.243542    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:18:28.280910 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:28.280872    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:18:29.323409 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:29.323344    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 02 16:18:31.410367 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.410335    2577 generic.go:358] "Generic (PLEG): container finished" podID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerID="472ee33f47e27403c1c1ea46a95c72ead9207edafdded2133a4997e7278bc96b" exitCode=0
Jun 02 16:18:31.410717 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.410375    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" event={"ID":"0572d6e5-fdf0-4d96-a572-cc8fb5d51548","Type":"ContainerDied","Data":"472ee33f47e27403c1c1ea46a95c72ead9207edafdded2133a4997e7278bc96b"}
Jun 02 16:18:31.536451 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.536427    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:18:31.670144 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.670047    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-openshift-service-ca-bundle\") pod \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\" (UID: \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\") "
Jun 02 16:18:31.670144 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.670108    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-proxy-tls\") pod \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\" (UID: \"0572d6e5-fdf0-4d96-a572-cc8fb5d51548\") "
Jun 02 16:18:31.670432 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.670380    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "0572d6e5-fdf0-4d96-a572-cc8fb5d51548" (UID: "0572d6e5-fdf0-4d96-a572-cc8fb5d51548"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:18:31.672163 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.672138    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0572d6e5-fdf0-4d96-a572-cc8fb5d51548" (UID: "0572d6e5-fdf0-4d96-a572-cc8fb5d51548"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:18:31.771716 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.771674    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:18:31.771716 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:31.771709    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0572d6e5-fdf0-4d96-a572-cc8fb5d51548-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:18:32.415330 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:32.415296    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2" event={"ID":"0572d6e5-fdf0-4d96-a572-cc8fb5d51548","Type":"ContainerDied","Data":"3ef72f126fd7f3afb5cb164e7e44cb09b0bec23ef0ee3f5369814877aaa1f0a9"}
Jun 02 16:18:32.415330 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:32.415329    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"
Jun 02 16:18:32.415837 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:32.415343    2577 scope.go:117] "RemoveContainer" containerID="472ee33f47e27403c1c1ea46a95c72ead9207edafdded2133a4997e7278bc96b"
Jun 02 16:18:32.436041 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:32.435994    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"]
Jun 02 16:18:32.448620 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:32.448589    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-022f7-65bb66f8cc-9hzd2"]
Jun 02 16:18:34.222368 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:34.222338    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" path="/var/lib/kubelet/pods/0572d6e5-fdf0-4d96-a572-cc8fb5d51548/volumes"
Jun 02 16:18:36.844710 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.844671    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"]
Jun 02 16:18:36.845193 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845164    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf"
Jun 02 16:18:36.845193 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845182    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf"
Jun 02 16:18:36.845308 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845205    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7"
Jun 02 16:18:36.845308 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845215    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7"
Jun 02 16:18:36.845308 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845241    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container"
Jun 02 16:18:36.845308 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845250    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container"
Jun 02 16:18:36.845308 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845262    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kube-rbac-proxy"
Jun 02 16:18:36.845308 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845270    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kube-rbac-proxy"
Jun 02 16:18:36.845638 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845347    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kserve-container"
Jun 02 16:18:36.845638 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845360    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="0572d6e5-fdf0-4d96-a572-cc8fb5d51548" containerName="splitter-graph-022f7"
Jun 02 16:18:36.845638 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845374    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="55d5560a-75f7-446b-a1cb-49388f257cef" containerName="kube-rbac-proxy"
Jun 02 16:18:36.845638 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.845408    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="7e5f20e9-0a9b-4ae9-a989-045dcba3cdaf" containerName="sequence-graph-411bf"
Jun 02 16:18:36.849114 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.849093    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:36.850827 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.850807    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-78d4f-serving-cert\""
Jun 02 16:18:36.850921 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.850867    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-78d4f-kube-rbac-proxy-sar-config\""
Jun 02 16:18:36.854358 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:36.854333    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"]
Jun 02 16:18:37.035343 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.021491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d6e7e4-f726-44bf-b58b-770cab906329-openshift-service-ca-bundle\") pod \"switch-graph-78d4f-7578756596-fctpv\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") " pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:37.035343 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.021583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls\") pod \"switch-graph-78d4f-7578756596-fctpv\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") " pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:37.123058 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.122975    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d6e7e4-f726-44bf-b58b-770cab906329-openshift-service-ca-bundle\") pod \"switch-graph-78d4f-7578756596-fctpv\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") " pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:37.123195 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.123055    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls\") pod \"switch-graph-78d4f-7578756596-fctpv\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") " pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:37.123238 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:18:37.123207    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-78d4f-serving-cert: secret "switch-graph-78d4f-serving-cert" not found
Jun 02 16:18:37.123299 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:18:37.123287    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls podName:d1d6e7e4-f726-44bf-b58b-770cab906329 nodeName:}" failed. No retries permitted until 2026-06-02 16:18:37.623270798 +0000 UTC m=+1921.965691997 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls") pod "switch-graph-78d4f-7578756596-fctpv" (UID: "d1d6e7e4-f726-44bf-b58b-770cab906329") : secret "switch-graph-78d4f-serving-cert" not found
Jun 02 16:18:37.123687 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.123670    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d6e7e4-f726-44bf-b58b-770cab906329-openshift-service-ca-bundle\") pod \"switch-graph-78d4f-7578756596-fctpv\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") " pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:37.626735 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.626695    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls\") pod \"switch-graph-78d4f-7578756596-fctpv\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") " pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:37.629165 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.629132    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls\") pod \"switch-graph-78d4f-7578756596-fctpv\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") " pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:37.760265 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.760228    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:37.891992 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:37.891838    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"]
Jun 02 16:18:37.894629 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:18:37.894601    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d6e7e4_f726_44bf_b58b_770cab906329.slice/crio-9141da6db6ed5d7af22a4cfa1a13844dcc7fb4ea4e53f7310b9314e6a533e0ce WatchSource:0}: Error finding container 9141da6db6ed5d7af22a4cfa1a13844dcc7fb4ea4e53f7310b9314e6a533e0ce: Status 404 returned error can't find the container with id 9141da6db6ed5d7af22a4cfa1a13844dcc7fb4ea4e53f7310b9314e6a533e0ce
Jun 02 16:18:38.436997 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:38.436955    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" event={"ID":"d1d6e7e4-f726-44bf-b58b-770cab906329","Type":"ContainerStarted","Data":"b2efbc6ac998ab103206972aaa7c9cc187e3990c15169ea009cc45f89fb018ac"}
Jun 02 16:18:38.436997 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:38.436992    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" event={"ID":"d1d6e7e4-f726-44bf-b58b-770cab906329","Type":"ContainerStarted","Data":"9141da6db6ed5d7af22a4cfa1a13844dcc7fb4ea4e53f7310b9314e6a533e0ce"}
Jun 02 16:18:38.437261 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:38.437018    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:38.453797 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:38.453750    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" podStartSLOduration=2.453734778 podStartE2EDuration="2.453734778s" podCreationTimestamp="2026-06-02 16:18:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:18:38.452383803 +0000 UTC m=+1922.794805025" watchObservedRunningTime="2026-06-02 16:18:38.453734778 +0000 UTC m=+1922.796156000"
Jun 02 16:18:39.323574 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:39.323533    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 02 16:18:44.445651 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:44.445619    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:18:49.323520 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:18:49.323490    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:19:01.104321 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.104276    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"]
Jun 02 16:19:01.108202 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.108181    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:01.110123 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.110098    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-cb302-serving-cert\""
Jun 02 16:19:01.110123 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.110115    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-cb302-kube-rbac-proxy-sar-config\""
Jun 02 16:19:01.116850 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.116828    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"]
Jun 02 16:19:01.228472 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.228437    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-openshift-service-ca-bundle\") pod \"splitter-graph-cb302-6d88566858-jszmf\" (UID: \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\") " pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:01.228657 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.228503    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-proxy-tls\") pod \"splitter-graph-cb302-6d88566858-jszmf\" (UID: \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\") " pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:01.329406 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.329349    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-proxy-tls\") pod \"splitter-graph-cb302-6d88566858-jszmf\" (UID: \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\") " pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:01.329573 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.329472    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-openshift-service-ca-bundle\") pod \"splitter-graph-cb302-6d88566858-jszmf\" (UID: \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\") " pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:01.330081 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.330064    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-openshift-service-ca-bundle\") pod \"splitter-graph-cb302-6d88566858-jszmf\" (UID: \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\") " pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:01.331764 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.331743    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-proxy-tls\") pod \"splitter-graph-cb302-6d88566858-jszmf\" (UID: \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\") " pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:01.418878 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.418843    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:01.556611 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:01.556578    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"]
Jun 02 16:19:02.512071 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:02.512024    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" event={"ID":"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f","Type":"ContainerStarted","Data":"bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f"}
Jun 02 16:19:02.512071 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:02.512066    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" event={"ID":"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f","Type":"ContainerStarted","Data":"e49f15ee3df042b06de6101774b0aa3e0c33cbfc335bac2a0e08aaab58d82992"}
Jun 02 16:19:02.512592 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:02.512151    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:19:02.530739 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:02.530679    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" podStartSLOduration=1.530661155 podStartE2EDuration="1.530661155s" podCreationTimestamp="2026-06-02 16:19:01 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:19:02.529304091 +0000 UTC m=+1946.871725313" watchObservedRunningTime="2026-06-02 16:19:02.530661155 +0000 UTC m=+1946.873082380"
Jun 02 16:19:08.521532 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:19:08.521500    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:27:15.822570 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:15.822492    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"]
Jun 02 16:27:15.823079 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:15.822732    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302" containerID="cri-o://bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f" gracePeriod=30
Jun 02 16:27:15.957937 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:15.957902    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"]
Jun 02 16:27:15.958254 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:15.958205    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container" containerID="cri-o://4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13" gracePeriod=30
Jun 02 16:27:15.958415 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:15.958206    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kube-rbac-proxy" containerID="cri-o://5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb" gracePeriod=30
Jun 02 16:27:16.998786 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:16.998754    2577 generic.go:358] "Generic (PLEG): container finished" podID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerID="5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb" exitCode=2
Jun 02 16:27:16.999177 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:16.998818    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" event={"ID":"b0c59584-eaab-4fd0-b6af-fb75dd0abef1","Type":"ContainerDied","Data":"5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb"}
Jun 02 16:27:18.519981 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:18.519940    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:27:19.204151 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.204125    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:27:19.346866 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.346769    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xr652\" (UniqueName: \"kubernetes.io/projected/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-kube-api-access-xr652\") pod \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") "
Jun 02 16:27:19.347033 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.346868    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-cb302-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-error-404-isvc-cb302-kube-rbac-proxy-sar-config\") pod \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") "
Jun 02 16:27:19.347033 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.346913    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-proxy-tls\") pod \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\" (UID: \"b0c59584-eaab-4fd0-b6af-fb75dd0abef1\") "
Jun 02 16:27:19.347806 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.347778    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-error-404-isvc-cb302-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-cb302-kube-rbac-proxy-sar-config") pod "b0c59584-eaab-4fd0-b6af-fb75dd0abef1" (UID: "b0c59584-eaab-4fd0-b6af-fb75dd0abef1"). InnerVolumeSpecName "error-404-isvc-cb302-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:27:19.349108 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.349086    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0c59584-eaab-4fd0-b6af-fb75dd0abef1" (UID: "b0c59584-eaab-4fd0-b6af-fb75dd0abef1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:27:19.349199 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.349144    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-kube-api-access-xr652" (OuterVolumeSpecName: "kube-api-access-xr652") pod "b0c59584-eaab-4fd0-b6af-fb75dd0abef1" (UID: "b0c59584-eaab-4fd0-b6af-fb75dd0abef1"). InnerVolumeSpecName "kube-api-access-xr652". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:27:19.447914 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.447880    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-cb302-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-error-404-isvc-cb302-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:27:19.447914 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.447909    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:27:19.447914 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:19.447919    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr652\" (UniqueName: \"kubernetes.io/projected/b0c59584-eaab-4fd0-b6af-fb75dd0abef1-kube-api-access-xr652\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:27:20.008994 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.008960    2577 generic.go:358] "Generic (PLEG): container finished" podID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerID="4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13" exitCode=0
Jun 02 16:27:20.009414 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.009035    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" event={"ID":"b0c59584-eaab-4fd0-b6af-fb75dd0abef1","Type":"ContainerDied","Data":"4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13"}
Jun 02 16:27:20.009414 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.009042    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"
Jun 02 16:27:20.009414 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.009062    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg" event={"ID":"b0c59584-eaab-4fd0-b6af-fb75dd0abef1","Type":"ContainerDied","Data":"f42df99fedd04401b757a2963e459399b51cd897967a14c8aa0b22af4c688913"}
Jun 02 16:27:20.009414 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.009077    2577 scope.go:117] "RemoveContainer" containerID="5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb"
Jun 02 16:27:20.026079 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.026056    2577 scope.go:117] "RemoveContainer" containerID="4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13"
Jun 02 16:27:20.033896 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.033875    2577 scope.go:117] "RemoveContainer" containerID="5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb"
Jun 02 16:27:20.034179 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:27:20.034149    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb\": container with ID starting with 5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb not found: ID does not exist" containerID="5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb"
Jun 02 16:27:20.034245 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.034194    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb"} err="failed to get container status \"5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb\": rpc error: code = NotFound desc = could not find container \"5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb\": container with ID starting with 5871405ef1914ffcb3031a3a56e5476b5c0d098513005450842e8fa2691a0beb not found: ID does not exist"
Jun 02 16:27:20.034245 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.034223    2577 scope.go:117] "RemoveContainer" containerID="4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13"
Jun 02 16:27:20.034541 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:27:20.034519    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13\": container with ID starting with 4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13 not found: ID does not exist" containerID="4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13"
Jun 02 16:27:20.034603 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.034550    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13"} err="failed to get container status \"4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13\": rpc error: code = NotFound desc = could not find container \"4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13\": container with ID starting with 4411ffec301305a5b8db5f6e2fcb1a847581611791b2dfe2cd696f0ab32f3f13 not found: ID does not exist"
Jun 02 16:27:20.041503 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.041380    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"]
Jun 02 16:27:20.043480 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:27:20.043460    2577 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c59584_eaab_4fd0_b6af_fb75dd0abef1.slice\": RecentStats: unable to find data in memory cache]"
Jun 02 16:27:20.043541 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:27:20.043497    2577 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c59584_eaab_4fd0_b6af_fb75dd0abef1.slice/crio-f42df99fedd04401b757a2963e459399b51cd897967a14c8aa0b22af4c688913\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c59584_eaab_4fd0_b6af_fb75dd0abef1.slice\": RecentStats: unable to find data in memory cache]"
Jun 02 16:27:20.047405 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.047368    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cb302-predictor-6f58d6db6-sp4rg"]
Jun 02 16:27:20.222788 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:20.222756    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" path="/var/lib/kubelet/pods/b0c59584-eaab-4fd0-b6af-fb75dd0abef1/volumes"
Jun 02 16:27:23.519661 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:23.519620    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:27:28.520426 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:28.520371    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:27:28.520835 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:28.520491    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:27:33.519664 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:33.519623    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:27:38.520289 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:38.520250    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:27:43.520482 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:43.520438    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:27:45.970259 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:45.970233    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:27:46.074813 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.074786    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-openshift-service-ca-bundle\") pod \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\" (UID: \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\") "
Jun 02 16:27:46.074981 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.074840    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-proxy-tls\") pod \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\" (UID: \"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f\") "
Jun 02 16:27:46.075148 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.075123    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" (UID: "1e921bb4-bfe3-4a13-99ca-8d0aa04f906f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:27:46.076778 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.076728    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" (UID: "1e921bb4-bfe3-4a13-99ca-8d0aa04f906f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:27:46.085519 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.085491    2577 generic.go:358] "Generic (PLEG): container finished" podID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerID="bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f" exitCode=0
Jun 02 16:27:46.085663 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.085542    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" event={"ID":"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f","Type":"ContainerDied","Data":"bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f"}
Jun 02 16:27:46.085663 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.085556    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"
Jun 02 16:27:46.085663 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.085574    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf" event={"ID":"1e921bb4-bfe3-4a13-99ca-8d0aa04f906f","Type":"ContainerDied","Data":"e49f15ee3df042b06de6101774b0aa3e0c33cbfc335bac2a0e08aaab58d82992"}
Jun 02 16:27:46.085663 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.085593    2577 scope.go:117] "RemoveContainer" containerID="bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f"
Jun 02 16:27:46.095594 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.095352    2577 scope.go:117] "RemoveContainer" containerID="bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f"
Jun 02 16:27:46.095806 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:27:46.095782    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f\": container with ID starting with bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f not found: ID does not exist" containerID="bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f"
Jun 02 16:27:46.095904 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.095815    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f"} err="failed to get container status \"bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f\": rpc error: code = NotFound desc = could not find container \"bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f\": container with ID starting with bef8370209823b825ca15e927ee5b0c9ddec4054331084e1b12c4f57a4dbc27f not found: ID does not exist"
Jun 02 16:27:46.108016 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.107992    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"]
Jun 02 16:27:46.111541 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.111521    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-cb302-6d88566858-jszmf"]
Jun 02 16:27:46.175674 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.175640    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:27:46.175674 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.175669    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:27:46.224879 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:27:46.224853    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" path="/var/lib/kubelet/pods/1e921bb4-bfe3-4a13-99ca-8d0aa04f906f/volumes"
Jun 02 16:34:56.137551 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:56.137472    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"]
Jun 02 16:34:56.140106 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:56.137721    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f" containerID="cri-o://b2efbc6ac998ab103206972aaa7c9cc187e3990c15169ea009cc45f89fb018ac" gracePeriod=30
Jun 02 16:34:56.340563 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:56.340532    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"]
Jun 02 16:34:56.340838 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:56.340807    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container" containerID="cri-o://bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98" gracePeriod=30
Jun 02 16:34:56.340949 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:56.340856    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kube-rbac-proxy" containerID="cri-o://ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5" gracePeriod=30
Jun 02 16:34:57.356610 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:57.356568    2577 generic.go:358] "Generic (PLEG): container finished" podID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerID="ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5" exitCode=2
Jun 02 16:34:57.357003 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:57.356641    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" event={"ID":"5cf88d9e-57a2-42ce-b89b-f63ef0df9719","Type":"ContainerDied","Data":"ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5"}
Jun 02 16:34:58.066011 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.065975    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-chk7k/must-gather-gp9wf"]
Jun 02 16:34:58.066330 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066318    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302"
Jun 02 16:34:58.066371 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066332    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302"
Jun 02 16:34:58.066371 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066344    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container"
Jun 02 16:34:58.066371 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066350    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container"
Jun 02 16:34:58.066501 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066372    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kube-rbac-proxy"
Jun 02 16:34:58.066501 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066382    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kube-rbac-proxy"
Jun 02 16:34:58.066501 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066467    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="1e921bb4-bfe3-4a13-99ca-8d0aa04f906f" containerName="splitter-graph-cb302"
Jun 02 16:34:58.066501 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066478    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kube-rbac-proxy"
Jun 02 16:34:58.066501 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.066484    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0c59584-eaab-4fd0-b6af-fb75dd0abef1" containerName="kserve-container"
Jun 02 16:34:58.069490 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.069473    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:34:58.071605 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.071584    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-chk7k\"/\"openshift-service-ca.crt\""
Jun 02 16:34:58.071724 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.071628    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-chk7k\"/\"default-dockercfg-q52nq\""
Jun 02 16:34:58.072079 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.072065    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-chk7k\"/\"kube-root-ca.crt\""
Jun 02 16:34:58.080976 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.080953    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-chk7k/must-gather-gp9wf"]
Jun 02 16:34:58.140686 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.140647    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/02de6d1d-83b9-4029-bedc-240b77f812a2-must-gather-output\") pod \"must-gather-gp9wf\" (UID: \"02de6d1d-83b9-4029-bedc-240b77f812a2\") " pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:34:58.140686 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.140691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn6kd\" (UniqueName: \"kubernetes.io/projected/02de6d1d-83b9-4029-bedc-240b77f812a2-kube-api-access-gn6kd\") pod \"must-gather-gp9wf\" (UID: \"02de6d1d-83b9-4029-bedc-240b77f812a2\") " pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:34:58.241785 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.241749    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/02de6d1d-83b9-4029-bedc-240b77f812a2-must-gather-output\") pod \"must-gather-gp9wf\" (UID: \"02de6d1d-83b9-4029-bedc-240b77f812a2\") " pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:34:58.241785 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.241784    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gn6kd\" (UniqueName: \"kubernetes.io/projected/02de6d1d-83b9-4029-bedc-240b77f812a2-kube-api-access-gn6kd\") pod \"must-gather-gp9wf\" (UID: \"02de6d1d-83b9-4029-bedc-240b77f812a2\") " pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:34:58.242071 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.242053    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/02de6d1d-83b9-4029-bedc-240b77f812a2-must-gather-output\") pod \"must-gather-gp9wf\" (UID: \"02de6d1d-83b9-4029-bedc-240b77f812a2\") " pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:34:58.250435 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.250411    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn6kd\" (UniqueName: \"kubernetes.io/projected/02de6d1d-83b9-4029-bedc-240b77f812a2-kube-api-access-gn6kd\") pod \"must-gather-gp9wf\" (UID: \"02de6d1d-83b9-4029-bedc-240b77f812a2\") " pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:34:58.398570 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.398537    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:34:58.533646 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.533598    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-chk7k/must-gather-gp9wf"]
Jun 02 16:34:58.540179 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:34:58.540148    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02de6d1d_83b9_4029_bedc_240b77f812a2.slice/crio-9880d1bc0438d7cd25543367c5f701023300c7cb92832dc2e8dff0d6a8d967da WatchSource:0}: Error finding container 9880d1bc0438d7cd25543367c5f701023300c7cb92832dc2e8dff0d6a8d967da: Status 404 returned error can't find the container with id 9880d1bc0438d7cd25543367c5f701023300c7cb92832dc2e8dff0d6a8d967da
Jun 02 16:34:58.542119 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:58.542102    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 02 16:34:59.364338 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.364299    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chk7k/must-gather-gp9wf" event={"ID":"02de6d1d-83b9-4029-bedc-240b77f812a2","Type":"ContainerStarted","Data":"9880d1bc0438d7cd25543367c5f701023300c7cb92832dc2e8dff0d6a8d967da"}
Jun 02 16:34:59.447292 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.447010    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:34:59.623254 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.622892    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:34:59.754405 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.754348    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-proxy-tls\") pod \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") "
Jun 02 16:34:59.754593 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.754448    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp7lr\" (UniqueName: \"kubernetes.io/projected/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-kube-api-access-vp7lr\") pod \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") "
Jun 02 16:34:59.754593 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.754514    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"error-404-isvc-78d4f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-error-404-isvc-78d4f-kube-rbac-proxy-sar-config\") pod \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\" (UID: \"5cf88d9e-57a2-42ce-b89b-f63ef0df9719\") "
Jun 02 16:34:59.754985 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.754949    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-error-404-isvc-78d4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "error-404-isvc-78d4f-kube-rbac-proxy-sar-config") pod "5cf88d9e-57a2-42ce-b89b-f63ef0df9719" (UID: "5cf88d9e-57a2-42ce-b89b-f63ef0df9719"). InnerVolumeSpecName "error-404-isvc-78d4f-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:34:59.756853 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.756825    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-kube-api-access-vp7lr" (OuterVolumeSpecName: "kube-api-access-vp7lr") pod "5cf88d9e-57a2-42ce-b89b-f63ef0df9719" (UID: "5cf88d9e-57a2-42ce-b89b-f63ef0df9719"). InnerVolumeSpecName "kube-api-access-vp7lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:34:59.756960 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.756865    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5cf88d9e-57a2-42ce-b89b-f63ef0df9719" (UID: "5cf88d9e-57a2-42ce-b89b-f63ef0df9719"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:34:59.855601 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.855516    2577 reconciler_common.go:299] "Volume detached for volume \"error-404-isvc-78d4f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-error-404-isvc-78d4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:34:59.855601 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.855558    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:34:59.855601 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:34:59.855576    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp7lr\" (UniqueName: \"kubernetes.io/projected/5cf88d9e-57a2-42ce-b89b-f63ef0df9719-kube-api-access-vp7lr\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:35:00.369843 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.369798    2577 generic.go:358] "Generic (PLEG): container finished" podID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerID="bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98" exitCode=0
Jun 02 16:35:00.370039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.369890    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"
Jun 02 16:35:00.370039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.369927    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" event={"ID":"5cf88d9e-57a2-42ce-b89b-f63ef0df9719","Type":"ContainerDied","Data":"bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98"}
Jun 02 16:35:00.370039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.369955    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb" event={"ID":"5cf88d9e-57a2-42ce-b89b-f63ef0df9719","Type":"ContainerDied","Data":"f94a92e71e3285c1e7b796551b205098f6d81f1bc80c81c95e49545f574a6090"}
Jun 02 16:35:00.370039 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.369974    2577 scope.go:117] "RemoveContainer" containerID="ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5"
Jun 02 16:35:00.382662 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.382634    2577 scope.go:117] "RemoveContainer" containerID="bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98"
Jun 02 16:35:00.388259 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.388228    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"]
Jun 02 16:35:00.393354 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.393331    2577 scope.go:117] "RemoveContainer" containerID="ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5"
Jun 02 16:35:00.393851 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:35:00.393824    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5\": container with ID starting with ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5 not found: ID does not exist" containerID="ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5"
Jun 02 16:35:00.393953 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.393864    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5"} err="failed to get container status \"ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5\": rpc error: code = NotFound desc = could not find container \"ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5\": container with ID starting with ce924d17cff7f9aaeb372d14171b5d8f2af9eb4064f4c61712ca8182379ffdb5 not found: ID does not exist"
Jun 02 16:35:00.393953 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.393916    2577 scope.go:117] "RemoveContainer" containerID="bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98"
Jun 02 16:35:00.394193 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:35:00.394169    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98\": container with ID starting with bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98 not found: ID does not exist" containerID="bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98"
Jun 02 16:35:00.394252 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.394204    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98"} err="failed to get container status \"bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98\": rpc error: code = NotFound desc = could not find container \"bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98\": container with ID starting with bd3be31322fd58e3623f1a9f22d4942ca92080327625165b4d79dc82af7b4c98 not found: ID does not exist"
Jun 02 16:35:00.395036 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:00.395013    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-78d4f-predictor-79467495b8-z9tfb"]
Jun 02 16:35:02.224591 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:02.224554    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" path="/var/lib/kubelet/pods/5cf88d9e-57a2-42ce-b89b-f63ef0df9719/volumes"
Jun 02 16:35:04.444959 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:04.444898    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:35:05.389089 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:05.389050    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chk7k/must-gather-gp9wf" event={"ID":"02de6d1d-83b9-4029-bedc-240b77f812a2","Type":"ContainerStarted","Data":"922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b"}
Jun 02 16:35:05.389089 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:05.389092    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chk7k/must-gather-gp9wf" event={"ID":"02de6d1d-83b9-4029-bedc-240b77f812a2","Type":"ContainerStarted","Data":"8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64"}
Jun 02 16:35:05.405247 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:05.405179    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-chk7k/must-gather-gp9wf" podStartSLOduration=1.066785591 podStartE2EDuration="7.40516028s" podCreationTimestamp="2026-06-02 16:34:58 +0000 UTC" firstStartedPulling="2026-06-02 16:34:58.542220628 +0000 UTC m=+2902.884641828" lastFinishedPulling="2026-06-02 16:35:04.880595296 +0000 UTC m=+2909.223016517" observedRunningTime="2026-06-02 16:35:05.404098405 +0000 UTC m=+2909.746519628" watchObservedRunningTime="2026-06-02 16:35:05.40516028 +0000 UTC m=+2909.747581505"
Jun 02 16:35:09.444683 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:09.444641    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:35:09.445097 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:09.444748    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:35:14.007674 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:14.007631    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:14.445990 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:14.445948    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:35:14.843901 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:14.843827    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:15.670200 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:15.670164    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:16.500719 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:16.500676    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:17.359380 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:17.359347    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:18.187472 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:18.187439    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:19.008558 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:19.008529    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:19.445276 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:19.445223    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:35:19.853800 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:19.853720    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:20.681606 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:20.681577    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:21.505400 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:21.505353    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:22.337252 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:22.337212    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:23.198585 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:23.198554    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-78d4f-7578756596-fctpv_d1d6e7e4-f726-44bf-b58b-770cab906329/switch-graph-78d4f/0.log"
Jun 02 16:35:24.444357 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:24.444318    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 02 16:35:26.461104 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.461071    2577 generic.go:358] "Generic (PLEG): container finished" podID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerID="8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64" exitCode=0
Jun 02 16:35:26.461585 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.461142    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-chk7k/must-gather-gp9wf" event={"ID":"02de6d1d-83b9-4029-bedc-240b77f812a2","Type":"ContainerDied","Data":"8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64"}
Jun 02 16:35:26.461585 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.461548    2577 scope.go:117] "RemoveContainer" containerID="8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64"
Jun 02 16:35:26.462300 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.462280    2577 generic.go:358] "Generic (PLEG): container finished" podID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerID="b2efbc6ac998ab103206972aaa7c9cc187e3990c15169ea009cc45f89fb018ac" exitCode=0
Jun 02 16:35:26.462359 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.462322    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" event={"ID":"d1d6e7e4-f726-44bf-b58b-770cab906329","Type":"ContainerDied","Data":"b2efbc6ac998ab103206972aaa7c9cc187e3990c15169ea009cc45f89fb018ac"}
Jun 02 16:35:26.780924 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.780900    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:35:26.802241 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.802210    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d6e7e4-f726-44bf-b58b-770cab906329-openshift-service-ca-bundle\") pod \"d1d6e7e4-f726-44bf-b58b-770cab906329\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") "
Jun 02 16:35:26.802435 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.802263    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls\") pod \"d1d6e7e4-f726-44bf-b58b-770cab906329\" (UID: \"d1d6e7e4-f726-44bf-b58b-770cab906329\") "
Jun 02 16:35:26.802620 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.802587    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d6e7e4-f726-44bf-b58b-770cab906329-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d1d6e7e4-f726-44bf-b58b-770cab906329" (UID: "d1d6e7e4-f726-44bf-b58b-770cab906329"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 02 16:35:26.804340 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.804310    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1d6e7e4-f726-44bf-b58b-770cab906329" (UID: "d1d6e7e4-f726-44bf-b58b-770cab906329"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 02 16:35:26.902933 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.902893    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1d6e7e4-f726-44bf-b58b-770cab906329-openshift-service-ca-bundle\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:35:26.902933 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:26.902931    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1d6e7e4-f726-44bf-b58b-770cab906329-proxy-tls\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:35:27.185443 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.185408    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-chk7k_must-gather-gp9wf_02de6d1d-83b9-4029-bedc-240b77f812a2/gather/0.log"
Jun 02 16:35:27.466596 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.466564    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv" event={"ID":"d1d6e7e4-f726-44bf-b58b-770cab906329","Type":"ContainerDied","Data":"9141da6db6ed5d7af22a4cfa1a13844dcc7fb4ea4e53f7310b9314e6a533e0ce"}
Jun 02 16:35:27.466596 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.466590    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"
Jun 02 16:35:27.467036 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.466614    2577 scope.go:117] "RemoveContainer" containerID="b2efbc6ac998ab103206972aaa7c9cc187e3990c15169ea009cc45f89fb018ac"
Jun 02 16:35:27.488243 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.488215    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"]
Jun 02 16:35:27.492896 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.492871    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-78d4f-7578756596-fctpv"]
Jun 02 16:35:27.881290 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881203    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pm69g/must-gather-p6cz6"]
Jun 02 16:35:27.881610 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881595    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container"
Jun 02 16:35:27.881659 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881612    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container"
Jun 02 16:35:27.881659 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881630    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f"
Jun 02 16:35:27.881659 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881636    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f"
Jun 02 16:35:27.881659 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881646    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kube-rbac-proxy"
Jun 02 16:35:27.881659 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881652    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kube-rbac-proxy"
Jun 02 16:35:27.881815 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881707    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kserve-container"
Jun 02 16:35:27.881815 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881717    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="5cf88d9e-57a2-42ce-b89b-f63ef0df9719" containerName="kube-rbac-proxy"
Jun 02 16:35:27.881815 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.881724    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" containerName="switch-graph-78d4f"
Jun 02 16:35:27.884423 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.884385    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pm69g/must-gather-p6cz6"
Jun 02 16:35:27.886373 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.886341    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pm69g\"/\"kube-root-ca.crt\""
Jun 02 16:35:27.886518 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.886374    2577 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pm69g\"/\"openshift-service-ca.crt\""
Jun 02 16:35:27.886518 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.886346    2577 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pm69g\"/\"default-dockercfg-sjn2p\""
Jun 02 16:35:27.893974 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.893951    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pm69g/must-gather-p6cz6"]
Jun 02 16:35:27.910416 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.910374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d4b88ae-4d8e-4863-a49e-ecf8d4a48518-must-gather-output\") pod \"must-gather-p6cz6\" (UID: \"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518\") " pod="openshift-must-gather-pm69g/must-gather-p6cz6"
Jun 02 16:35:27.910558 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:27.910467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnd2\" (UniqueName: \"kubernetes.io/projected/0d4b88ae-4d8e-4863-a49e-ecf8d4a48518-kube-api-access-jrnd2\") pod \"must-gather-p6cz6\" (UID: \"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518\") " pod="openshift-must-gather-pm69g/must-gather-p6cz6"
Jun 02 16:35:28.010974 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:28.010931    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d4b88ae-4d8e-4863-a49e-ecf8d4a48518-must-gather-output\") pod \"must-gather-p6cz6\" (UID: \"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518\") " pod="openshift-must-gather-pm69g/must-gather-p6cz6"
Jun 02 16:35:28.011190 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:28.010994    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnd2\" (UniqueName: \"kubernetes.io/projected/0d4b88ae-4d8e-4863-a49e-ecf8d4a48518-kube-api-access-jrnd2\") pod \"must-gather-p6cz6\" (UID: \"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518\") " pod="openshift-must-gather-pm69g/must-gather-p6cz6"
Jun 02 16:35:28.011314 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:28.011292    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0d4b88ae-4d8e-4863-a49e-ecf8d4a48518-must-gather-output\") pod \"must-gather-p6cz6\" (UID: \"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518\") " pod="openshift-must-gather-pm69g/must-gather-p6cz6"
Jun 02 16:35:28.018959 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:28.018935    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnd2\" (UniqueName: \"kubernetes.io/projected/0d4b88ae-4d8e-4863-a49e-ecf8d4a48518-kube-api-access-jrnd2\") pod \"must-gather-p6cz6\" (UID: \"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518\") " pod="openshift-must-gather-pm69g/must-gather-p6cz6"
Jun 02 16:35:28.194161 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:28.194122    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pm69g/must-gather-p6cz6"
Jun 02 16:35:28.223507 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:28.223477    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d6e7e4-f726-44bf-b58b-770cab906329" path="/var/lib/kubelet/pods/d1d6e7e4-f726-44bf-b58b-770cab906329/volumes"
Jun 02 16:35:28.322938 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:28.322906    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pm69g/must-gather-p6cz6"]
Jun 02 16:35:28.326512 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:35:28.326487    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d4b88ae_4d8e_4863_a49e_ecf8d4a48518.slice/crio-1666fc6c0387927d295dbbd1ce6a46c5a0258ab620890179516cca80aaeaba20 WatchSource:0}: Error finding container 1666fc6c0387927d295dbbd1ce6a46c5a0258ab620890179516cca80aaeaba20: Status 404 returned error can't find the container with id 1666fc6c0387927d295dbbd1ce6a46c5a0258ab620890179516cca80aaeaba20
Jun 02 16:35:28.471331 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:28.471243    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pm69g/must-gather-p6cz6" event={"ID":"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518","Type":"ContainerStarted","Data":"1666fc6c0387927d295dbbd1ce6a46c5a0258ab620890179516cca80aaeaba20"}
Jun 02 16:35:29.477106 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:29.477062    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pm69g/must-gather-p6cz6" event={"ID":"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518","Type":"ContainerStarted","Data":"9d9f50d8389aef13373897e6e7c7368f3119d5245d48a8fc9bfbc8e811846011"}
Jun 02 16:35:29.477510 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:29.477111    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pm69g/must-gather-p6cz6" event={"ID":"0d4b88ae-4d8e-4863-a49e-ecf8d4a48518","Type":"ContainerStarted","Data":"48464c1f113e2c452e39ef3084d48e1c5179c17448ece7dd58a26aecbd13a3db"}
Jun 02 16:35:29.493284 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:29.493223    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-pm69g/must-gather-p6cz6" podStartSLOduration=1.5580086789999998 podStartE2EDuration="2.493205604s" podCreationTimestamp="2026-06-02 16:35:27 +0000 UTC" firstStartedPulling="2026-06-02 16:35:28.328181919 +0000 UTC m=+2932.670603120" lastFinishedPulling="2026-06-02 16:35:29.263378845 +0000 UTC m=+2933.605800045" observedRunningTime="2026-06-02 16:35:29.492309627 +0000 UTC m=+2933.834730850" watchObservedRunningTime="2026-06-02 16:35:29.493205604 +0000 UTC m=+2933.835626829"
Jun 02 16:35:30.863734 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:30.863702    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x5r6b_eae3712a-0e98-45de-a57a-3f0f8ae67289/global-pull-secret-syncer/0.log"
Jun 02 16:35:30.959871 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:30.959837    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bb8nn_07285198-8458-4168-854e-d4df187aec14/konnectivity-agent/0.log"
Jun 02 16:35:31.047562 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:31.047532    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-168.ec2.internal_72c56ec27649d987d63e9d06a174d926/haproxy/0.log"
Jun 02 16:35:32.730789 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:32.730743    2577 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-chk7k/must-gather-gp9wf"]
Jun 02 16:35:32.731403 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:32.731356    2577 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-chk7k/must-gather-gp9wf" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerName="copy" containerID="cri-o://922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b" gracePeriod=2
Jun 02 16:35:32.741416 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:32.736709    2577 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-chk7k/must-gather-gp9wf"]
Jun 02 16:35:33.133594 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.133543    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-chk7k_must-gather-gp9wf_02de6d1d-83b9-4029-bedc-240b77f812a2/copy/0.log"
Jun 02 16:35:33.136446 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.134007    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:35:33.137297 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.137258    2577 status_manager.go:895] "Failed to get status for pod" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" pod="openshift-must-gather-chk7k/must-gather-gp9wf" err="pods \"must-gather-gp9wf\" is forbidden: User \"system:node:ip-10-0-133-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-chk7k\": no relationship found between node 'ip-10-0-133-168.ec2.internal' and this object"
Jun 02 16:35:33.162920 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.160642    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn6kd\" (UniqueName: \"kubernetes.io/projected/02de6d1d-83b9-4029-bedc-240b77f812a2-kube-api-access-gn6kd\") pod \"02de6d1d-83b9-4029-bedc-240b77f812a2\" (UID: \"02de6d1d-83b9-4029-bedc-240b77f812a2\") "
Jun 02 16:35:33.162920 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.160752    2577 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/02de6d1d-83b9-4029-bedc-240b77f812a2-must-gather-output\") pod \"02de6d1d-83b9-4029-bedc-240b77f812a2\" (UID: \"02de6d1d-83b9-4029-bedc-240b77f812a2\") "
Jun 02 16:35:33.162920 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.162618    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02de6d1d-83b9-4029-bedc-240b77f812a2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "02de6d1d-83b9-4029-bedc-240b77f812a2" (UID: "02de6d1d-83b9-4029-bedc-240b77f812a2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 02 16:35:33.176804 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.176745    2577 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02de6d1d-83b9-4029-bedc-240b77f812a2-kube-api-access-gn6kd" (OuterVolumeSpecName: "kube-api-access-gn6kd") pod "02de6d1d-83b9-4029-bedc-240b77f812a2" (UID: "02de6d1d-83b9-4029-bedc-240b77f812a2"). InnerVolumeSpecName "kube-api-access-gn6kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 02 16:35:33.262461 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.262376    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/02de6d1d-83b9-4029-bedc-240b77f812a2-must-gather-output\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:35:33.262461 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.262431    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn6kd\" (UniqueName: \"kubernetes.io/projected/02de6d1d-83b9-4029-bedc-240b77f812a2-kube-api-access-gn6kd\") on node \"ip-10-0-133-168.ec2.internal\" DevicePath \"\""
Jun 02 16:35:33.492976 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.492948    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-chk7k_must-gather-gp9wf_02de6d1d-83b9-4029-bedc-240b77f812a2/copy/0.log"
Jun 02 16:35:33.493717 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.493688    2577 generic.go:358] "Generic (PLEG): container finished" podID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerID="922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b" exitCode=143
Jun 02 16:35:33.493970 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.493956    2577 scope.go:117] "RemoveContainer" containerID="922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b"
Jun 02 16:35:33.494235 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.494217    2577 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-chk7k/must-gather-gp9wf"
Jun 02 16:35:33.511488 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.511376    2577 status_manager.go:895] "Failed to get status for pod" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" pod="openshift-must-gather-chk7k/must-gather-gp9wf" err="pods \"must-gather-gp9wf\" is forbidden: User \"system:node:ip-10-0-133-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-chk7k\": no relationship found between node 'ip-10-0-133-168.ec2.internal' and this object"
Jun 02 16:35:33.513889 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.513862    2577 status_manager.go:895] "Failed to get status for pod" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" pod="openshift-must-gather-chk7k/must-gather-gp9wf" err="pods \"must-gather-gp9wf\" is forbidden: User \"system:node:ip-10-0-133-168.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-chk7k\": no relationship found between node 'ip-10-0-133-168.ec2.internal' and this object"
Jun 02 16:35:33.531216 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.531096    2577 scope.go:117] "RemoveContainer" containerID="8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64"
Jun 02 16:35:33.565145 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.565116    2577 scope.go:117] "RemoveContainer" containerID="922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b"
Jun 02 16:35:33.568837 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:35:33.568796    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b\": container with ID starting with 922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b not found: ID does not exist" containerID="922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b"
Jun 02 16:35:33.568990 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.568843    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b"} err="failed to get container status \"922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b\": rpc error: code = NotFound desc = could not find container \"922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b\": container with ID starting with 922c95340d4e2f481076e6f06d405d0d4afb988c695d8843878387a993992d9b not found: ID does not exist"
Jun 02 16:35:33.568990 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.568872    2577 scope.go:117] "RemoveContainer" containerID="8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64"
Jun 02 16:35:33.569235 ip-10-0-133-168 kubenswrapper[2577]: E0602 16:35:33.569205    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64\": container with ID starting with 8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64 not found: ID does not exist" containerID="8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64"
Jun 02 16:35:33.569322 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:33.569243    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64"} err="failed to get container status \"8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64\": rpc error: code = NotFound desc = could not find container \"8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64\": container with ID starting with 8643c33aa8f06ae5c768cdf392aec694b1876e8392f2df33f06581cc6ce7ee64 not found: ID does not exist"
Jun 02 16:35:34.225587 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.225548    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" path="/var/lib/kubelet/pods/02de6d1d-83b9-4029-bedc-240b77f812a2/volumes"
Jun 02 16:35:34.487337 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.487268    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e4c2e8b9-46e7-4320-99a0-89a78f98ed27/alertmanager/0.log"
Jun 02 16:35:34.516591 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.516559    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e4c2e8b9-46e7-4320-99a0-89a78f98ed27/config-reloader/0.log"
Jun 02 16:35:34.543605 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.543540    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e4c2e8b9-46e7-4320-99a0-89a78f98ed27/kube-rbac-proxy-web/0.log"
Jun 02 16:35:34.575639 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.575608    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e4c2e8b9-46e7-4320-99a0-89a78f98ed27/kube-rbac-proxy/0.log"
Jun 02 16:35:34.603816 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.603750    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e4c2e8b9-46e7-4320-99a0-89a78f98ed27/kube-rbac-proxy-metric/0.log"
Jun 02 16:35:34.633858 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.632807    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e4c2e8b9-46e7-4320-99a0-89a78f98ed27/prom-label-proxy/0.log"
Jun 02 16:35:34.661019 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.660979    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_e4c2e8b9-46e7-4320-99a0-89a78f98ed27/init-config-reloader/0.log"
Jun 02 16:35:34.743261 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.743182    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-sgxtg_fa081bae-80b7-4e42-bb44-69311dbb44cc/kube-state-metrics/0.log"
Jun 02 16:35:34.767694 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.767663    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-sgxtg_fa081bae-80b7-4e42-bb44-69311dbb44cc/kube-rbac-proxy-main/0.log"
Jun 02 16:35:34.798407 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.798355    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-8469c486bb-sgxtg_fa081bae-80b7-4e42-bb44-69311dbb44cc/kube-rbac-proxy-self/0.log"
Jun 02 16:35:34.834754 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.834722    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67ddbc6856-ws4wh_e6a477b5-895e-4688-ad78-cb2c0ee2422e/metrics-server/0.log"
Jun 02 16:35:34.865632 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.865598    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5768fcc9dd-cdxhj_cb3c660f-a1d0-4f45-9cfe-3c704c609c5e/monitoring-plugin/0.log"
Jun 02 16:35:34.923545 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.923517    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fqrt_2dfe68d2-3070-49c4-a420-e5b9dd76322c/node-exporter/0.log"
Jun 02 16:35:34.969924 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.969898    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fqrt_2dfe68d2-3070-49c4-a420-e5b9dd76322c/kube-rbac-proxy/0.log"
Jun 02 16:35:34.996983 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:34.996904    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fqrt_2dfe68d2-3070-49c4-a420-e5b9dd76322c/init-textfile/0.log"
Jun 02 16:35:35.199686 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.199659    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-rvlz9_cf51054f-3332-4b3c-8396-67be19d8bb24/kube-rbac-proxy-main/0.log"
Jun 02 16:35:35.226861 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.226833    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-rvlz9_cf51054f-3332-4b3c-8396-67be19d8bb24/kube-rbac-proxy-self/0.log"
Jun 02 16:35:35.251875 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.251789    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-85658796db-rvlz9_cf51054f-3332-4b3c-8396-67be19d8bb24/openshift-state-metrics/0.log"
Jun 02 16:35:35.301124 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.301088    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9898e745-053c-4cdf-89b8-a0a93dad3b8d/prometheus/0.log"
Jun 02 16:35:35.320446 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.320415    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9898e745-053c-4cdf-89b8-a0a93dad3b8d/config-reloader/0.log"
Jun 02 16:35:35.344687 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.344661    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9898e745-053c-4cdf-89b8-a0a93dad3b8d/thanos-sidecar/0.log"
Jun 02 16:35:35.370609 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.370580    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9898e745-053c-4cdf-89b8-a0a93dad3b8d/kube-rbac-proxy-web/0.log"
Jun 02 16:35:35.396680 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.396649    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9898e745-053c-4cdf-89b8-a0a93dad3b8d/kube-rbac-proxy/0.log"
Jun 02 16:35:35.421087 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.421056    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9898e745-053c-4cdf-89b8-a0a93dad3b8d/kube-rbac-proxy-thanos/0.log"
Jun 02 16:35:35.446316 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.446285    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9898e745-053c-4cdf-89b8-a0a93dad3b8d/init-config-reloader/0.log"
Jun 02 16:35:35.529033 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.528952    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-59b7f8756b-m2h8s_ab578e0a-45c7-4be1-ac08-8fa0559acb54/prometheus-operator-admission-webhook/0.log"
Jun 02 16:35:35.571831 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.571795    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc5d86cb8-b8vrp_0e1a51e0-f52e-4d54-8d67-dad858c300de/telemeter-client/0.log"
Jun 02 16:35:35.606465 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.606433    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc5d86cb8-b8vrp_0e1a51e0-f52e-4d54-8d67-dad858c300de/reload/0.log"
Jun 02 16:35:35.641711 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:35.641677    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6dc5d86cb8-b8vrp_0e1a51e0-f52e-4d54-8d67-dad858c300de/kube-rbac-proxy/0.log"
Jun 02 16:35:37.858938 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:37.858904    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-c65b8dc4b-l8vns_3fc9bebc-7481-4323-9202-4f0a72ac4151/download-server/0.log"
Jun 02 16:35:38.641326 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.641285    2577 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"]
Jun 02 16:35:38.641834 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.641809    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerName="gather"
Jun 02 16:35:38.641997 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.641984    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerName="gather"
Jun 02 16:35:38.642093 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.642081    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerName="copy"
Jun 02 16:35:38.642184 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.642174    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerName="copy"
Jun 02 16:35:38.642405 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.642377    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerName="gather"
Jun 02 16:35:38.642521 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.642510    2577 memory_manager.go:356] "RemoveStaleState removing state" podUID="02de6d1d-83b9-4029-bedc-240b77f812a2" containerName="copy"
Jun 02 16:35:38.647402 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.647377    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.659651 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.659621    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"]
Jun 02 16:35:38.713857 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.713794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-sys\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.714076 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.713872    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-lib-modules\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.714076 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.713934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5ld\" (UniqueName: \"kubernetes.io/projected/d24017e7-19f0-41e5-a359-997a55734718-kube-api-access-hw5ld\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.714076 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.713966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-proc\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.714076 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.713996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-podres\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814504 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814446    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5ld\" (UniqueName: \"kubernetes.io/projected/d24017e7-19f0-41e5-a359-997a55734718-kube-api-access-hw5ld\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814504 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814507    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-proc\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814734 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814575    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-proc\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814734 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814668    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-podres\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814803 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814751    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-sys\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814803 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814788    2577 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-lib-modules\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814803 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814799    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-podres\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814897 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814832    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-sys\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.814931 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.814898    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d24017e7-19f0-41e5-a359-997a55734718-lib-modules\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.823455 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.823432    2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5ld\" (UniqueName: \"kubernetes.io/projected/d24017e7-19f0-41e5-a359-997a55734718-kube-api-access-hw5ld\") pod \"perf-node-gather-daemonset-tm6kk\" (UID: \"d24017e7-19f0-41e5-a359-997a55734718\") " pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:38.956933 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:38.956901    2577 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:39.100244 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.100206    2577 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"]
Jun 02 16:35:39.103558 ip-10-0-133-168 kubenswrapper[2577]: W0602 16:35:39.103530    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd24017e7_19f0_41e5_a359_997a55734718.slice/crio-b2dc96fe4f059332d650321dfe8c5829d3457a255a70ff6b780ef7e2d07a11fc WatchSource:0}: Error finding container b2dc96fe4f059332d650321dfe8c5829d3457a255a70ff6b780ef7e2d07a11fc: Status 404 returned error can't find the container with id b2dc96fe4f059332d650321dfe8c5829d3457a255a70ff6b780ef7e2d07a11fc
Jun 02 16:35:39.160863 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.160834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jllqh_84dc34aa-0d48-4060-a21d-c93066f8c683/dns/0.log"
Jun 02 16:35:39.183505 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.183473    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jllqh_84dc34aa-0d48-4060-a21d-c93066f8c683/kube-rbac-proxy/0.log"
Jun 02 16:35:39.210594 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.210513    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2zxzn_46bdfe7b-9361-4645-bea3-a81b1c280953/dns-node-resolver/0.log"
Jun 02 16:35:39.518277 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.518182    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk" event={"ID":"d24017e7-19f0-41e5-a359-997a55734718","Type":"ContainerStarted","Data":"a8d597bf34d5ab8a3d84cb8a7882d85e2ede0552bfc30eee0cdc4defa3bd4d78"}
Jun 02 16:35:39.518277 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.518216    2577 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk" event={"ID":"d24017e7-19f0-41e5-a359-997a55734718","Type":"ContainerStarted","Data":"b2dc96fe4f059332d650321dfe8c5829d3457a255a70ff6b780ef7e2d07a11fc"}
Jun 02 16:35:39.518558 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.518277    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:39.542900 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.542813    2577 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk" podStartSLOduration=1.542792519 podStartE2EDuration="1.542792519s" podCreationTimestamp="2026-06-02 16:35:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-06-02 16:35:39.541007729 +0000 UTC m=+2943.883428958" watchObservedRunningTime="2026-06-02 16:35:39.542792519 +0000 UTC m=+2943.885213741"
Jun 02 16:35:39.813155 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:39.813032    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mjvd5_497b35e9-18f2-40e6-939d-50cd84bf8ad4/node-ca/0.log"
Jun 02 16:35:40.999232 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:40.999202    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-d6xhm_3384c3b6-30a7-4de5-86d6-a139400ecaf6/serve-healthcheck-canary/0.log"
Jun 02 16:35:41.619768 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:41.619732    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tl27z_84cb8848-73cd-40a0-902a-11fc078cff29/kube-rbac-proxy/0.log"
Jun 02 16:35:41.645503 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:41.645475    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tl27z_84cb8848-73cd-40a0-902a-11fc078cff29/exporter/0.log"
Jun 02 16:35:41.672708 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:41.672678    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-tl27z_84cb8848-73cd-40a0-902a-11fc078cff29/extractor/0.log"
Jun 02 16:35:43.916410 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:43.916362    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-2dxcd_61707792-f927-4b6e-9216-00fe50a3d306/s3-init/0.log"
Jun 02 16:35:43.948020 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:43.947983    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-79f457656b-bp58r_fb67966a-cb68-40d4-8b42-f97c928c540e/seaweedfs/0.log"
Jun 02 16:35:45.537229 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:45.537203    2577 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pm69g/perf-node-gather-daemonset-tm6kk"
Jun 02 16:35:48.165223 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:48.165190    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-96c6f749-zgp74_8418f950-f099-4d7a-9991-1a13c03d241a/migrator/0.log"
Jun 02 16:35:48.188242 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:48.188216    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-96c6f749-zgp74_8418f950-f099-4d7a-9991-1a13c03d241a/graceful-termination/0.log"
Jun 02 16:35:49.606485 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:49.606454    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6pv8_906be90d-c7ef-421a-ab64-22e1e9ef50c1/kube-multus-additional-cni-plugins/0.log"
Jun 02 16:35:49.633600 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:49.633571    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6pv8_906be90d-c7ef-421a-ab64-22e1e9ef50c1/egress-router-binary-copy/0.log"
Jun 02 16:35:49.657575 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:49.657543    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6pv8_906be90d-c7ef-421a-ab64-22e1e9ef50c1/cni-plugins/0.log"
Jun 02 16:35:49.693414 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:49.693369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6pv8_906be90d-c7ef-421a-ab64-22e1e9ef50c1/bond-cni-plugin/0.log"
Jun 02 16:35:49.724291 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:49.724245    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6pv8_906be90d-c7ef-421a-ab64-22e1e9ef50c1/routeoverride-cni/0.log"
Jun 02 16:35:49.760192 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:49.760166    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6pv8_906be90d-c7ef-421a-ab64-22e1e9ef50c1/whereabouts-cni-bincopy/0.log"
Jun 02 16:35:49.792947 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:49.792913    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-c6pv8_906be90d-c7ef-421a-ab64-22e1e9ef50c1/whereabouts-cni/0.log"
Jun 02 16:35:50.297195 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:50.297125    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hq8fg_e1c64c4e-eb38-4c9b-b944-719fe01bb403/kube-multus/0.log"
Jun 02 16:35:50.455455 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:50.455429    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mdgjl_f13e68f4-6625-437f-bb2c-974057826f5c/network-metrics-daemon/0.log"
Jun 02 16:35:50.474974 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:50.474950    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mdgjl_f13e68f4-6625-437f-bb2c-974057826f5c/kube-rbac-proxy/0.log"
Jun 02 16:35:52.121575 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:52.121540    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xq8p6_e15ec370-ab69-4d86-9aa5-23bbe054cfc0/ovn-controller/0.log"
Jun 02 16:35:52.173005 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:52.172964    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xq8p6_e15ec370-ab69-4d86-9aa5-23bbe054cfc0/ovn-acl-logging/0.log"
Jun 02 16:35:52.195619 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:52.195586    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xq8p6_e15ec370-ab69-4d86-9aa5-23bbe054cfc0/kube-rbac-proxy-node/0.log"
Jun 02 16:35:52.220865 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:52.220830    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xq8p6_e15ec370-ab69-4d86-9aa5-23bbe054cfc0/kube-rbac-proxy-ovn-metrics/0.log"
Jun 02 16:35:52.244637 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:52.244606    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xq8p6_e15ec370-ab69-4d86-9aa5-23bbe054cfc0/northd/0.log"
Jun 02 16:35:52.267689 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:52.267664    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xq8p6_e15ec370-ab69-4d86-9aa5-23bbe054cfc0/nbdb/0.log"
Jun 02 16:35:52.293587 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:52.293557    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xq8p6_e15ec370-ab69-4d86-9aa5-23bbe054cfc0/sbdb/0.log"
Jun 02 16:35:52.503122 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:52.503090    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-xq8p6_e15ec370-ab69-4d86-9aa5-23bbe054cfc0/ovnkube-controller/0.log"
Jun 02 16:35:53.646219 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:53.646185    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wp2zq_91fef626-9a99-4972-8f0f-59ee18f994ec/network-check-target-container/0.log"
Jun 02 16:35:54.636781 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:54.636755    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8zpqk_053754bb-d8a8-43c7-a4b4-3c8db21c5095/iptables-alerter/0.log"
Jun 02 16:35:55.464299 ip-10-0-133-168 kubenswrapper[2577]: I0602 16:35:55.464224    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-s6d6g_2b7bc71d-feea-4854-98c8-fddb75f27a89/tuned/0.log"