Jun 15 05:45:14.239564 ip-10-0-133-111 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jun 15 05:45:14.239574 ip-10-0-133-111 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jun 15 05:45:14.239581 ip-10-0-133-111 systemd[1]: kubelet.service: Failed with result 'resources'.
Jun 15 05:45:14.239821 ip-10-0-133-111 systemd[1]: Failed to start Kubernetes Kubelet.
Jun 15 05:45:24.263710 ip-10-0-133-111 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jun 15 05:45:24.263726 ip-10-0-133-111 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ce59909b918b4d809b47688acfd0da07 --
Jun 15 05:47:36.927696 ip-10-0-133-111 systemd[1]: Starting Kubernetes Kubelet...
Jun 15 05:47:37.329268 ip-10-0-133-111 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:37.329268 ip-10-0-133-111 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 15 05:47:37.329268 ip-10-0-133-111 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:37.329268 ip-10-0-133-111 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 15 05:47:37.329268 ip-10-0-133-111 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:47:37.332363 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.332277    2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 15 05:47:37.338709 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338694    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:37.338709 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338709    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338713    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338716    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338719    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338721    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338724    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338727    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338729    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338733    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338737    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338739    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338742    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338745    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338748    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338751    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338753    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338756    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338759    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338762    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338764    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:37.338781 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338772    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338775    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338778    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338780    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338783    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338786    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338791    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338795    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338798    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338801    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338804    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338807    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338810    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338813    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338816    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338818    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338821    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338823    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338826    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338828    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:37.339248 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338831    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338833    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338836    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338838    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338841    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338843    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338846    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338848    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338850    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338853    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338856    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338860    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338864    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338867    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338871    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338873    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338876    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338879    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338882    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:37.339779 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338884    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338887    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338889    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338891    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338894    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338896    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338899    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338901    2562 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338904    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338906    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338909    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338911    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338914    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338916    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338920    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338923    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338926    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338929    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338933    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:37.340251 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338936    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338938    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338941    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338943    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338946    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338949    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338951    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338953    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338957    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338960    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338963    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338966    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338969    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338971    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338974    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338976    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338979    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338981    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338983    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:37.340737 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.338986    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:37.341649 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341636    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:37.341682 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341650    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:37.341682 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341654    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:37.341682 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341658    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:37.341682 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341661    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:37.341682 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341664    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:37.341682 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341684    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341688    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341691    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341694    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341697    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341699    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341702    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341705    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341707    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341710    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341712    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341715    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341717    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341720    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341722    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341725    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341728    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341730    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341733    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:37.341840 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341735    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341737    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341740    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341743    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341745    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341747    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341750    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341752    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341755    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341757    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341759    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341762    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341764    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341767    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341771    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341775    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341779    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341782    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341784    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:37.342338 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341787    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341789    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341792    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341794    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341797    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341799    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341802    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341804    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341807    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341810    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341813    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341815    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341818    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341820    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341823    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341827    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341830    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341833    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341836    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341838    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:37.342820 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341841    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341844    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341846    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341849    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341852    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341854    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341857    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341859    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341863    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341866    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341868    2562 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341871    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341873    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341876    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341880    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341883    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341885    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341888    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341890    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341893    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:37.343591 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341895    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341898    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341900    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341903    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341906    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341909    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341911    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341914    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341916    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341919    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341921    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341924    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341927    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341929    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.341932    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342334    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342339    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342343    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342346    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342349    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:37.344134 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342353    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342355    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342358    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342361    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342364    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342366    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342369    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342371    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342374    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342391    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342395    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342397    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342400    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342403    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342405    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342409    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342411    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342414    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342416    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342419    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:37.344646 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342421    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342424    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342427    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342429    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342432    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342435    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342438    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342440    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342443    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342445    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342448    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342450    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342453    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342455    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342458    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342460    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342463    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342467    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342470    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342473    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:37.345137 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342476    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342478    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342481    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342483    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342486    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342488    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342490    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342493    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342496    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342498    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342501    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342503    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342505    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342508    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342511    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342513    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342516    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342531    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342534    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342538    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:37.345631 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342540    2562 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342543    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342546    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342549    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342553    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342556    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342559    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342562    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342564    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342567    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342570    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342574    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342577    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342581    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342585    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342589    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342593    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342597    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342601    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:37.346127 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342604    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342607    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342611    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342614    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342617    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342619    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342622    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342625    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342627    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342630    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342633    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342636    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342638    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342641    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.342643    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342713    2562 flags.go:64] FLAG: --address="0.0.0.0"
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342724    2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342734    2562 flags.go:64] FLAG: --anonymous-auth="true"
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342740    2562 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342746    2562 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342750    2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 15 05:47:37.346617 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342754    2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342758    2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342761    2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342764    2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342768    2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342771    2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342774    2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342778    2562 flags.go:64] FLAG: --cgroup-root=""
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342781    2562 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342784    2562 flags.go:64] FLAG: --client-ca-file=""
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342787    2562 flags.go:64] FLAG: --cloud-provider="external"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342790    2562 flags.go:64] FLAG: --cluster-dns="[]"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342864    2562 flags.go:64] FLAG: --cluster-domain=""
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342870    2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342885    2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342891    2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342897    2562 flags.go:64] FLAG: --container-log-max-files="5"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342903    2562 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342908    2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342914    2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342929    2562 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342934    2562 flags.go:64] FLAG: --contention-profiling="false"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342939    2562 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.342945    2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 15 05:47:37.347110 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.343850    2562 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.343872    2562 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.343880    2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.343885    2562 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.343889    2562 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.343893    2562 flags.go:64] FLAG: --enable-load-reader="false"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.343897    2562 flags.go:64] FLAG: --enable-server="true"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.343900    2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344024    2562 flags.go:64] FLAG: --event-burst="100"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344031    2562 flags.go:64] FLAG: --event-qps="50"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344034    2562 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344038    2562 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344041    2562 flags.go:64] FLAG: --eviction-hard=""
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344047    2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344050    2562 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344054    2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344057    2562 flags.go:64] FLAG: --eviction-soft=""
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344060    2562 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344063    2562 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344067    2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344070    2562 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344073    2562 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344075    2562 flags.go:64] FLAG: --fail-swap-on="true"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344078    2562 flags.go:64] FLAG: --feature-gates=""
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344082    2562 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 15 05:47:37.347682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344085    2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344088    2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344091    2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344095    2562 flags.go:64] FLAG: --healthz-port="10248"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344098    2562 flags.go:64] FLAG: --help="false"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344100    2562 flags.go:64] FLAG: --hostname-override="ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344104    2562 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344107    2562 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344110    2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344114    2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344118    2562 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344121    2562 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344124    2562 flags.go:64] FLAG: --image-service-endpoint=""
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344126    2562 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344129    2562 flags.go:64] FLAG: --kube-api-burst="100"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344136    2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344139    2562 flags.go:64] FLAG: --kube-api-qps="50"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344142    2562 flags.go:64] FLAG: --kube-reserved=""
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344146    2562 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344149    2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344152    2562 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344155    2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344157    2562 flags.go:64] FLAG: --lock-file=""
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344160    2562 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 15 05:47:37.348340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344163    2562 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344166    2562 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344178    2562 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344181    2562 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344184    2562 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344187    2562 flags.go:64] FLAG: --logging-format="text"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344189    2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344193    2562 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344196    2562 flags.go:64] FLAG: --manifest-url=""
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344199    2562 flags.go:64] FLAG: --manifest-url-header=""
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344204    2562 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344207    2562 flags.go:64] FLAG: --max-open-files="1000000"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344211    2562 flags.go:64] FLAG: --max-pods="110"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344214    2562 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344217    2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344220    2562 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344223    2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344226    2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344229    2562 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344232    2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344239    2562 flags.go:64] FLAG: --node-status-max-images="50"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344242    2562 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344245    2562 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 15 05:47:37.348993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344249    2562 flags.go:64] FLAG: --pod-cidr=""
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344252    2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a2e90c8ef7d783e4d635e24fb09435f5e1f6e954583512c0944f9e57bc77b52"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344258    2562 flags.go:64] FLAG: --pod-manifest-path=""
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344261    2562 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344264    2562 flags.go:64] FLAG: --pods-per-core="0"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344267    2562 flags.go:64] FLAG: --port="10250"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344270    2562 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344273    2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0003c06a61f7848aa"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344277    2562 flags.go:64] FLAG: --qos-reserved=""
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344280    2562 flags.go:64] FLAG: --read-only-port="10255"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344283    2562 flags.go:64] FLAG: --register-node="true"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344286    2562 flags.go:64] FLAG: --register-with-taints=""
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344289    2562 flags.go:64] FLAG: --registry-burst="10"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344292    2562 flags.go:64] FLAG: --registry-qps="5"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344295    2562 flags.go:64] FLAG: --reserved-cpus=""
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344298    2562 flags.go:64] FLAG: --reserved-memory=""
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344301    2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344304    2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344307    2562 flags.go:64] FLAG: --rotate-certificates="false"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344310    2562 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344313    2562 flags.go:64] FLAG: --runonce="false"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344316    2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344319    2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344321    2562 flags.go:64] FLAG: --seccomp-default="false"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344324    2562 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 15 05:47:37.349584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344327    2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344330    2562 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344333    2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344336    2562 flags.go:64] FLAG: --storage-driver-password="root"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344339    2562 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344342    2562 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344345    2562 flags.go:64] FLAG: --storage-driver-user="root"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344347    2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344352    2562 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344355    2562 flags.go:64] FLAG: --system-cgroups=""
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344358    2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344364    2562 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344367    2562 flags.go:64] FLAG: --tls-cert-file=""
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344369    2562 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344373    2562 flags.go:64] FLAG: --tls-min-version=""
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344392    2562 flags.go:64] FLAG: --tls-private-key-file=""
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344396    2562 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344398    2562 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344401    2562 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344404    2562 flags.go:64] FLAG: --v="2"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344408    2562 flags.go:64] FLAG: --version="false"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344412    2562 flags.go:64] FLAG: --vmodule=""
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344416    2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.344419    2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 15 05:47:37.350183 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344539    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344543    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344546    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344549    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344552    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344555    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344557    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344560    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344564    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344567    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344571    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344574    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344577    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344587    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344592    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344595    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344601    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344604    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344606    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:37.350787 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344609    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344612    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344614    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344617    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344619    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344622    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344625    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344627    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344630    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344633    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344635    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344638    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344640    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344643    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344645    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344648    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344651    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344654    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344656    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:37.351243 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344659    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344661    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344664    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344667    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344670    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344673    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344676    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344679    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344682    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344684    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344688    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344691    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344695    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344699    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344703    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344706    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344710    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344713    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344716    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:37.351803 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344719    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344722    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344725    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344728    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344730    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344733    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344735    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344738    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344740    2562 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344743    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344745    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344748    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344750    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344753    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344755    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344758    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344760    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344764    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344766    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344769    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:37.352283 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344772    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344774    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344777    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344781    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344784    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344787    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344789    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344792    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344794    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344797    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344799    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344802    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344804    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344807    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344809    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344812    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344814    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344816    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344819    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344822    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:37.352824 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344825    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.344827    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.345581    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.352304    2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.352435    2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.352458    2562 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.352467    2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352523    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352528    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352532    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352535    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352539    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352543    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352546    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:37.353310 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352549    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352551    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352554    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352557    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352559    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352562    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352565    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352567    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352570    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352573    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352576    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352578    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352589    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352593    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352595    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352598    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352600    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352603    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352606    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352608    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:37.353676 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352611    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352613    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352618    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352621    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352624    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352626    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352629    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352633    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352637    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352640    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352643    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352646    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352649    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352652    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352654    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352657    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352659    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352662    2562 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352664    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352667    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:37.354157 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352669    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352672    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352674    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352677    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352679    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352682    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352684    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352686    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352689    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352692    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352694    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352697    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352699    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352702    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352705    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352708    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352711    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352714    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352716    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352719    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:37.354668 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352721    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352724    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352726    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352728    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352731    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352733    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352736    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352738    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352741    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352743    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352745    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352748    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352750    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352753    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352755    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352757    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352760    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352762    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352766    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352768    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:37.355151 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352771    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352773    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352776    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352778    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352781    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352784    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352786    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352789    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352792    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352795    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352797    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352800    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.352805    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352940    2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:47:37.355704 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352946    2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352949    2562 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352952    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352956    2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352959    2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352962    2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352965    2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352968    2562 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352970    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352972    2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352975    2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352978    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352980    2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352983    2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352985    2562 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352988    2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352990    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352993    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352995    2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.352998    2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:47:37.356094 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353000    2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353002    2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353005    2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353008    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353011    2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353014    2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353017    2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353020    2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353023    2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353026    2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353030    2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353033    2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353036    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353038    2562 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353041    2562 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353043    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353046    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353049    2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353051    2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353053    2562 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:47:37.356600 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353056    2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353058    2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353061    2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353063    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353065    2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353068    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353072    2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353075    2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353078    2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353080    2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353083    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353085    2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353088    2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353090    2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353093    2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353096    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353100    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353102    2562 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353105    2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:47:37.357086 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353107    2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353110    2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353112    2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353115    2562 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353117    2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353120    2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353122    2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353125    2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353127    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353129    2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353132    2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353135    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353137    2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353140    2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353142    2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353144    2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353147    2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353149    2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353152    2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:47:37.357563 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353154    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353158    2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353161    2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353164    2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353167    2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353170    2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353172    2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353175    2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353177    2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353180    2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353183    2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353186    2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353188    2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353191    2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353193    2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353197    2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353199    2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353202    2562 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353204    2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:47:37.358016 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:37.353207    2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:47:37.358503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.353211    2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:47:37.358503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.353324    2562 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 15 05:47:37.358503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.355360    2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 15 05:47:37.358503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.356712    2562 server.go:1019] "Starting client certificate rotation"
Jun 15 05:47:37.358503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.356814    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 15 05:47:37.358503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.356847    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 15 05:47:37.359481 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.359468    2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 15 05:47:37.360748 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.360727    2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 15 05:47:37.368460 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.368441    2562 log.go:25] "Validated CRI v1 runtime API"
Jun 15 05:47:37.372467 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.372453    2562 log.go:25] "Validated CRI v1 image API"
Jun 15 05:47:37.373412 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.373374    2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 15 05:47:37.377027 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.376997    2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b09ab7b5-4b1a-44f4-bbbe-734efbe268cb:/dev/nvme0n1p3 b7c8b8d7-ea43-4eb1-8c86-fd3b987a3079:/dev/nvme0n1p4]
Jun 15 05:47:37.377105 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.377026    2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 15 05:47:37.384221 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.384199    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 15 05:47:37.384897 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.384786    2562 manager.go:216] Machine: {Timestamp:2026-06-15 05:47:37.383439418 +0000 UTC m=+0.336306354 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3124150 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e97a93da92d366b099e69d93e473e SystemUUID:ec2e97a9-3da9-2d36-6b09-9e69d93e473e BootID:ce59909b-918b-4d80-9b47-688acfd0da07 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c8:46:f2:98:49 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c8:46:f2:98:49 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:be:73:3c:51:09 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 15 05:47:37.384897 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.384896    2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 15 05:47:37.385041 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.385029    2562 manager.go:232] Version: {KernelVersion:5.14.0-570.120.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260608-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 15 05:47:37.388437 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.388411    2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 15 05:47:37.388574 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.388441    2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 15 05:47:37.389125 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.389115    2562 topology_manager.go:138] "Creating topology manager with none policy"
Jun 15 05:47:37.389166 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.389127    2562 container_manager_linux.go:309] "Creating device plugin manager"
Jun 15 05:47:37.389166 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.389136    2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 15 05:47:37.389758 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.389747    2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 15 05:47:37.390945 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.390935    2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jun 15 05:47:37.393897 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.393885    2562 state_mem.go:36] "Initialized new in-memory state store"
Jun 15 05:47:37.394022 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.394013    2562 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jun 15 05:47:37.394060 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.394042    2562 kubelet.go:486] "Attempting to sync node with API server"
Jun 15 05:47:37.394060 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.394055    2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 15 05:47:37.394115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.394078    2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 15 05:47:37.394115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.394091    2562 kubelet.go:398] "Adding apiserver pod source"
Jun 15 05:47:37.394115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.394101    2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 15 05:47:37.394502 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.394490    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 15 05:47:37.394533 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.394510    2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 15 05:47:37.397970 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.397952    2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jun 15 05:47:37.399425 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.399412    2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 15 05:47:37.399475 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.399442    2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jun 15 05:47:37.399596 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.399587    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 15 05:47:37.399629 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.399601    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 15 05:47:37.399629 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.399607    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 15 05:47:37.399629 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.399612    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 15 05:47:37.399629 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.399619    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 15 05:47:37.399629 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.399625    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 15 05:47:37.400572 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.400558    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 15 05:47:37.400572 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.400568    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 15 05:47:37.400572 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.400576    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 15 05:47:37.400734 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.400593    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 15 05:47:37.400734 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.400602    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 15 05:47:37.400734 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.400609    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 15 05:47:37.401300 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.401291    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 15 05:47:37.401300 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.401299    2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 15 05:47:37.404493 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.404480    2562 server.go:1268] "Started kubelet"
Jun 15 05:47:37.404626 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.404555    2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 15 05:47:37.404685 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.404658    2562 server_v1.go:49] "podresources" method="list" useActivePods=true
Jun 15 05:47:37.405301 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.405107    2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 15 05:47:37.405301 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.405174    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 15 05:47:37.405301 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.405202    2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 15 05:47:37.405516 ip-10-0-133-111 systemd[1]: Started Kubernetes Kubelet.
Jun 15 05:47:37.405946 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.405517    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 15 05:47:37.406605 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.406572    2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 15 05:47:37.407452 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.407022    2562 server.go:310] "Adding debug handlers to kubelet server"
Jun 15 05:47:37.407554 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.407520    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6lqp5"
Jun 15 05:47:37.411123 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.410220    2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-111.ec2.internal.18b92b0bd413f27c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-111.ec2.internal,UID:ip-10-0-133-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-111.ec2.internal,},FirstTimestamp:2026-06-15 05:47:37.4044535 +0000 UTC m=+0.357320438,LastTimestamp:2026-06-15 05:47:37.4044535 +0000 UTC m=+0.357320438,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-111.ec2.internal,}"
Jun 15 05:47:37.411661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.411640    2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 15 05:47:37.411661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.411653    2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 15 05:47:37.411820 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.411757    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:37.412229 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.412203    2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jun 15 05:47:37.412324 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.412247    2562 volume_manager.go:311] "The desired_state_of_world populator starts"
Jun 15 05:47:37.412444 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.412433    2562 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jun 15 05:47:37.412644 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.412629    2562 reconstruct.go:97] "Volume reconstruction finished"
Jun 15 05:47:37.412644 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.412644    2562 reconciler.go:29] "Reconciler: start to sync state"
Jun 15 05:47:37.413006 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.412893    2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 15 05:47:37.413132 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.413116    2562 factory.go:153] Registering CRI-O factory
Jun 15 05:47:37.413169 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.413139    2562 factory.go:223] Registration of the crio container factory successfully
Jun 15 05:47:37.413217 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.413195    2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 15 05:47:37.413217 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.413209    2562 factory.go:55] Registering systemd factory
Jun 15 05:47:37.413217 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.413217    2562 factory.go:223] Registration of the systemd container factory successfully
Jun 15 05:47:37.413313 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.413240    2562 factory.go:103] Registering Raw factory
Jun 15 05:47:37.413313 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.413253    2562 manager.go:1196] Started watching for new ooms in manager
Jun 15 05:47:37.414124 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.414110    2562 manager.go:349] Starting recovery of all containers
Jun 15 05:47:37.416637 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.416603    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6lqp5"
Jun 15 05:47:37.421405 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.420102    2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jun 15 05:47:37.421405 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.420163    2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jun 15 05:47:37.426962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.426943    2562 manager.go:354] Recovery completed
Jun 15 05:47:37.431772 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.431756    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:37.433654 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.433636    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:37.433718 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.433667    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:37.433718 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.433676    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:37.434270 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.434253    2562 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 15 05:47:37.434270 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.434268    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:37.434471 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.434269    2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 15 05:47:37.434471 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.434330    2562 state_mem.go:36] "Initialized new in-memory state store"
Jun 15 05:47:37.435024 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.435008    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:37.435091 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.435039    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:37.435091 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.435053    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:37.435522 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.435511    2562 policy_none.go:49] "None policy: Start"
Jun 15 05:47:37.435572 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.435530    2562 memory_manager.go:187] "Starting memorymanager" policy="None"
Jun 15 05:47:37.435572 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.435542    2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jun 15 05:47:37.436457 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.436447    2562 policy_none.go:47] "Start"
Jun 15 05:47:37.468711 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.468697    2562 manager.go:337] "Starting Device Plugin manager"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.468759    2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.468773    2562 server.go:87] "Starting device plugin registration server"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.469037    2562 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.469053    2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.469232    2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.469369    2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.469397    2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.469896    2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.469936    2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.482014    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.483454    2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.483488    2562 status_manager.go:244] "Starting to sync pod status with apiserver"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.483506    2562 kubelet.go:2448] "Starting kubelet main sync loop"
Jun 15 05:47:37.485692 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.483570    2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 15 05:47:37.486441 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.485729    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:37.569866 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.569836    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:37.570841 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.570822    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:37.570925 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.570861    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:37.570925 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.570873    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:37.570925 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.570896    2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.577055 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.577039    2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.577102 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.577062    2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-111.ec2.internal\": node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:37.584189 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.584143    2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal"]
Jun 15 05:47:37.584238 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.584201    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:37.584948 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.584933    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:37.585013 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.584963    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:37.585013 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.584977    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:37.586350 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.586338    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:37.586495 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.586481    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.586548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.586507    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:37.586998 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.586974    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:37.586998 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.586994    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:37.586998 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.587004    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:37.587156 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.587023    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:37.587156 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.587046    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:37.587156 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.587055    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:37.588091 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.588070    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.588091 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.588096    2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:47:37.588739 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.588726    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:47:37.588808 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.588749    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:47:37.588808 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.588760    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:47:37.592359 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.592342    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:37.613978 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.613957    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08d00837dae608698233b66e665078bb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-111.ec2.internal\" (UID: \"08d00837dae608698233b66e665078bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.614072 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.613981    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dee957abc27887cd92f63b816a7fda6a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal\" (UID: \"dee957abc27887cd92f63b816a7fda6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.614072 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.613994    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-111.ec2.internal\" not found" node="ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.614072 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.614000    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee957abc27887cd92f63b816a7fda6a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal\" (UID: \"dee957abc27887cd92f63b816a7fda6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.618542 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.618524    2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-111.ec2.internal\" not found" node="ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.693012 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.692980    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:37.715043 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.715019    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee957abc27887cd92f63b816a7fda6a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal\" (UID: \"dee957abc27887cd92f63b816a7fda6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.715119 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.715048    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08d00837dae608698233b66e665078bb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-111.ec2.internal\" (UID: \"08d00837dae608698233b66e665078bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.715119 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.715066    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dee957abc27887cd92f63b816a7fda6a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal\" (UID: \"dee957abc27887cd92f63b816a7fda6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.715119 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.715099    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dee957abc27887cd92f63b816a7fda6a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal\" (UID: \"dee957abc27887cd92f63b816a7fda6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.715119 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.715110    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dee957abc27887cd92f63b816a7fda6a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal\" (UID: \"dee957abc27887cd92f63b816a7fda6a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.715265 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.715138    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/08d00837dae608698233b66e665078bb-config\") pod \"kube-apiserver-proxy-ip-10-0-133-111.ec2.internal\" (UID: \"08d00837dae608698233b66e665078bb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.794079 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.794043    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:37.894820 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.894743    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:37.916162 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.916133    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.921733 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:37.921712    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:37.995926 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:37.995882    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.096433 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.096401    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.197005 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.196916    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.297767 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.297611    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.356125 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.356086    2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 15 05:47:38.356832 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.356261    2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 15 05:47:38.398588 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.398564    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.402917 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:38.402887    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d00837dae608698233b66e665078bb.slice/crio-ac421811518326d859759b6ce88c57543e903aae19855df31b1ffda1044738ef WatchSource:0}: Error finding container ac421811518326d859759b6ce88c57543e903aae19855df31b1ffda1044738ef: Status 404 returned error can't find the container with id ac421811518326d859759b6ce88c57543e903aae19855df31b1ffda1044738ef
Jun 15 05:47:38.403228 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:38.403208    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee957abc27887cd92f63b816a7fda6a.slice/crio-ff13137399b152b14545214c448db5b34abd8e30cd683ce88cc9d6b3d84e8922 WatchSource:0}: Error finding container ff13137399b152b14545214c448db5b34abd8e30cd683ce88cc9d6b3d84e8922: Status 404 returned error can't find the container with id ff13137399b152b14545214c448db5b34abd8e30cd683ce88cc9d6b3d84e8922
Jun 15 05:47:38.408637 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.408623    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:47:38.421754 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.421729    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-14 05:42:37 +0000 UTC" deadline="2028-01-08 07:20:03.90066966 +0000 UTC"
Jun 15 05:47:38.421754 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.421754    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13729h32m25.478918678s"
Jun 15 05:47:38.486863 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.486813    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal" event={"ID":"08d00837dae608698233b66e665078bb","Type":"ContainerStarted","Data":"ac421811518326d859759b6ce88c57543e903aae19855df31b1ffda1044738ef"}
Jun 15 05:47:38.487816 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.487794    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal" event={"ID":"dee957abc27887cd92f63b816a7fda6a","Type":"ContainerStarted","Data":"ff13137399b152b14545214c448db5b34abd8e30cd683ce88cc9d6b3d84e8922"}
Jun 15 05:47:38.498965 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.498940    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.587905 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.587875    2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:38.599456 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.599432    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.699945 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.699898    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.734958 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.734936    2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:38.800085 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.799995    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.900998 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:38.900940    2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-111.ec2.internal\" not found"
Jun 15 05:47:38.958332 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:38.958166    2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:39.012191 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.012157    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:39.026323 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.026296    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 15 05:47:39.027593 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.027371    2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal"
Jun 15 05:47:39.036830 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.036801    2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 15 05:47:39.394839 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.394803    2562 apiserver.go:52] "Watching apiserver"
Jun 15 05:47:39.402968 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.402945    2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 15 05:47:39.405642 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.405616    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-4xgl2","openshift-multus/multus-additional-cni-plugins-kmskc","openshift-network-diagnostics/network-check-target-984wm","openshift-network-operator/iptables-alerter-hrdzf","openshift-ovn-kubernetes/ovnkube-node-rm8q8","kube-system/global-pull-secret-syncer-6hzm9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c","openshift-cluster-node-tuning-operator/tuned-7vb9c","openshift-dns/node-resolver-dzgqh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal","openshift-multus/multus-cl972","openshift-multus/network-metrics-daemon-cplw2","kube-system/konnectivity-agent-gbjb7","kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal"]
Jun 15 05:47:39.407390 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.407358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.408489 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.408475    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:47:39.409650 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.409632    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:39.409743 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.409700    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:39.410460 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.410440    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nqxck\""
Jun 15 05:47:39.410555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.410465    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 15 05:47:39.410555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.410489    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 15 05:47:39.410555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.410494    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 15 05:47:39.411288 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.411134    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 15 05:47:39.411288 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.411197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xvb9s\""
Jun 15 05:47:39.411466 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.411307    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.411615 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.411596    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 15 05:47:39.412136 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.412117    2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 15 05:47:39.412623 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.412608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.413559 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.413540    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lq6jc\""
Jun 15 05:47:39.414564 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.413995    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:47:39.414564 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.414192    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 15 05:47:39.414564 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.414357    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.414564 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.414428    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:39.414564 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.414516    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 15 05:47:39.415601 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.415573    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.418716 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.418693    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.420682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.420662    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 15 05:47:39.420792 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.420773    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 15 05:47:39.420792 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.420788    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 15 05:47:39.420940 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.420803    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 15 05:47:39.421082 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421062    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.421144 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421106    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 15 05:47:39.421144 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421109    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 15 05:47:39.421292 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421278    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8vkgw\""
Jun 15 05:47:39.421345 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421316    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 15 05:47:39.421452 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421062    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 15 05:47:39.421546 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421530    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6vbs\""
Jun 15 05:47:39.421723 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421709    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j44n5\""
Jun 15 05:47:39.421795 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.421720    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 15 05:47:39.422825 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.422808    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 15 05:47:39.422910 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.422815    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 15 05:47:39.423247 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.423233    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424622    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-env-overrides\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424659    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14e05b24-a058-4fef-b061-38cb4453e6c2-agent-certs\") pod \"konnectivity-agent-gbjb7\" (UID: \"14e05b24-a058-4fef-b061-38cb4453e6c2\") " pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424681    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd6fe0d-463d-4df8-a826-66922919a981-host-slash\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424699    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-etc-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424725    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-log-socket\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424747    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424767    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-run-netns\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424798    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovnkube-config\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovn-node-metrics-cert\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424853    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-device-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424886    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14e05b24-a058-4fef-b061-38cb4453e6c2-konnectivity-ca\") pod \"konnectivity-agent-gbjb7\" (UID: \"14e05b24-a058-4fef-b061-38cb4453e6c2\") " pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-systemd\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424918    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-cni-netd\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.424962 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424935    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-registration-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.424973    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425006    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-kubelet\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425030    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-var-lib-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425053    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425068    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-cni-bin\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425082    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvmw7\" (UniqueName: \"kubernetes.io/projected/9dd6fe0d-463d-4df8-a826-66922919a981-kube-api-access-lvmw7\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425098    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvgf9\" (UniqueName: \"kubernetes.io/projected/441e22c2-c489-4c53-aa0f-4d96208b16ff-kube-api-access-fvgf9\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425120    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-sys-fs\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425146    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvf7f\" (UniqueName: \"kubernetes.io/projected/f8ada3ca-4437-4389-908f-3c9d29d19ae8-kube-api-access-zvf7f\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovnkube-script-lib\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-socket-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425201    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425234    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-slash\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425291    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9dd6fe0d-463d-4df8-a826-66922919a981-iptables-alerter-script\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.425577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-systemd-units\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.426329 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425338    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-ovn\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.426329 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425362    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-node-log\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.426329 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.426329 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425547    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 15 05:47:39.426329 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425629    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.426329 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.425758    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hxqxt\""
Jun 15 05:47:39.426651 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.426635    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 15 05:47:39.426694 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.426662    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 15 05:47:39.426857 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.426836    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:39.426920 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.426904    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:39.427328 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.427312    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 15 05:47:39.427421 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.427408    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ldl5\""
Jun 15 05:47:39.427501 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.427484    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 15 05:47:39.427603 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.427581    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:47:39.427669 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.427312    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 15 05:47:39.428320 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.428301    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mhbb8\""
Jun 15 05:47:39.428421 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.428366    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 15 05:47:39.430623 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.430604    2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 15 05:47:39.447236 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.447217    2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wk6nf"
Jun 15 05:47:39.455719 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.455701    2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wk6nf"
Jun 15 05:47:39.513865 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.513840    2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jun 15 05:47:39.526223 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526195    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.526356 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526233    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-run-netns\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.526356 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526260    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovnkube-config\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.526356 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526310    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-os-release\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.526356 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526315    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-run-netns\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.526356 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526319    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.526356 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526337    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-device-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.526641 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526406    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.526641 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526421    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-device-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.526641 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526439    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-cni-multus\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.526641 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.526473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-registration-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.527120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527097    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovnkube-config\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.527225 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527209    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-kubelet\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.527278 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527243    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.527350 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527334    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.527423 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527405    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-kubelet\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.527471 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-cni-bin\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.527527 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527511    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysconfig\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.527577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527555    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-sys\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.527629 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527606    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-registration-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.527629 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527611    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgkt6\" (UniqueName: \"kubernetes.io/projected/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-kube-api-access-xgkt6\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.527726 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527687    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-lib-modules\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.527726 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527679    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-cni-bin\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.527726 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527721    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvmw7\" (UniqueName: \"kubernetes.io/projected/9dd6fe0d-463d-4df8-a826-66922919a981-kube-api-access-lvmw7\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.527903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527760    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvgf9\" (UniqueName: \"kubernetes.io/projected/441e22c2-c489-4c53-aa0f-4d96208b16ff-kube-api-access-fvgf9\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.527903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527794    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-kubernetes\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.527903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527830    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.527903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527861    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-system-cni-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.527903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527897    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-cni-bin\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.528134 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527941    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-kubelet\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.528134 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.527983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvf7f\" (UniqueName: \"kubernetes.io/projected/f8ada3ca-4437-4389-908f-3c9d29d19ae8-kube-api-access-zvf7f\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.528134 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528016    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovnkube-script-lib\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.528134 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528048    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/937dcb5e-3893-4ff2-a2f6-a3330a232700-kubelet-config\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.528312 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528181    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/937dcb5e-3893-4ff2-a2f6-a3330a232700-dbus\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.528446 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-cnibin\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.528543 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528460    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-os-release\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.528543 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528507    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-socket-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.528668 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528628    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-slash\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.528727 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528696    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-slash\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.528779 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528647    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-socket-dir\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.528779 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528741    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq8dw\" (UniqueName: \"kubernetes.io/projected/61f89985-9448-4de8-b773-dbeb59284d44-kube-api-access-wq8dw\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.528880 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-cnibin\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.528880 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-conf-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.528979 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528879    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-daemon-config\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.528979 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528905    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-etc-kubernetes\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.528979 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528939    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.528979 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.528972    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-node-log\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529165 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529005    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529165 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529032    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovnkube-script-lib\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529165 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f042f86e-21bb-4d7a-9141-e3fce2adadb3-tmp\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.529165 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529060    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-run-ovn-kubernetes\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529165 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529131    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-node-log\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-hosts-file\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.529408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529206    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.529408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529173    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529239    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-log-socket\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-multus-certs\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.529408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529306    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn2p6\" (UniqueName: \"kubernetes.io/projected/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-kube-api-access-tn2p6\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.529408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529298    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-log-socket\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529333    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovn-node-metrics-cert\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529431    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-var-lib-kubelet\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529473    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-host\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529515    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-system-cni-dir\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529554    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-socket-dir-parent\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529599    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-netns\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14e05b24-a058-4fef-b061-38cb4453e6c2-konnectivity-ca\") pod \"konnectivity-agent-gbjb7\" (UID: \"14e05b24-a058-4fef-b061-38cb4453e6c2\") " pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529658    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-systemd\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529686    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-cni-netd\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529727    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-cni-binary-copy\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.529763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529703    2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 15 05:47:39.530182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529784    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-hostroot\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.530182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529845    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:39.530182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529874    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-var-lib-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.530182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529902    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-modprobe-d\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.530182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.529945    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysctl-d\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.530182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysctl-conf\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.530182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530086    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-systemd\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.530182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530156    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-var-lib-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.530548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530092    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-host-cni-netd\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.530548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530227    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-tuned\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.530548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzdrp\" (UniqueName: \"kubernetes.io/projected/f042f86e-21bb-4d7a-9141-e3fce2adadb3-kube-api-access-pzdrp\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.530548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530341    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-run\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.530548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530417    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjt5f\" (UniqueName: \"kubernetes.io/projected/7139966e-e862-401e-bf2b-f13f4f334356-kube-api-access-kjt5f\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:39.530548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530449    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-sys-fs\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.530548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530507    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48938dcf-66f0-4920-be0c-f3735090e6c2-host\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.530548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530533    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48938dcf-66f0-4920-be0c-f3735090e6c2-serviceca\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.530888 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530642    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/14e05b24-a058-4fef-b061-38cb4453e6c2-konnectivity-ca\") pod \"konnectivity-agent-gbjb7\" (UID: \"14e05b24-a058-4fef-b061-38cb4453e6c2\") " pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:47:39.530888 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530708    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-systemd\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.530888 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530729    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-sys-fs\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.530888 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530736    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-tmp-dir\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.530888 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530851    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-cni-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.531115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530912    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.531115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530965    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9dd6fe0d-463d-4df8-a826-66922919a981-iptables-alerter-script\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.531115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.530979    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f8ada3ca-4437-4389-908f-3c9d29d19ae8-etc-selinux\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.531115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531002    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-systemd-units\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.531115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531046    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-systemd-units\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.531115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-ovn\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.531115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531090    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-env-overrides\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.531115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531096    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-run-ovn\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.531499 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531159    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.531499 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531286    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.531499 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531347    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-k8s-cni-cncf-io\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.531499 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531444    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14e05b24-a058-4fef-b061-38cb4453e6c2-agent-certs\") pod \"konnectivity-agent-gbjb7\" (UID: \"14e05b24-a058-4fef-b061-38cb4453e6c2\") " pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:47:39.531709 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531493    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd6fe0d-463d-4df8-a826-66922919a981-host-slash\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.531709 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531552    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-etc-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.531709 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531588    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22tfj\" (UniqueName: \"kubernetes.io/projected/48938dcf-66f0-4920-be0c-f3735090e6c2-kube-api-access-22tfj\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.531709 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531605    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/441e22c2-c489-4c53-aa0f-4d96208b16ff-env-overrides\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.531709 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531610    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9dd6fe0d-463d-4df8-a826-66922919a981-iptables-alerter-script\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.531709 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531665    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:39.531709 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531672    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/441e22c2-c489-4c53-aa0f-4d96208b16ff-etc-openvswitch\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.531709 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.531687    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9dd6fe0d-463d-4df8-a826-66922919a981-host-slash\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.534768 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.534748    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/441e22c2-c489-4c53-aa0f-4d96208b16ff-ovn-node-metrics-cert\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.534943 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.534921    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/14e05b24-a058-4fef-b061-38cb4453e6c2-agent-certs\") pod \"konnectivity-agent-gbjb7\" (UID: \"14e05b24-a058-4fef-b061-38cb4453e6c2\") " pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:47:39.538152 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.538129    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:39.538238 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.538156    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:39.538238 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.538169    2562 projected.go:196] Error preparing data for projected volume kube-api-access-8z9vj for pod openshift-network-diagnostics/network-check-target-984wm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:39.538357 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.538239    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj podName:c7c76571-ac25-4149-93e0-1c27217e11d1 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:40.038220249 +0000 UTC m=+2.991087192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8z9vj" (UniqueName: "kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj") pod "network-check-target-984wm" (UID: "c7c76571-ac25-4149-93e0-1c27217e11d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:39.539106 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.539080    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvmw7\" (UniqueName: \"kubernetes.io/projected/9dd6fe0d-463d-4df8-a826-66922919a981-kube-api-access-lvmw7\") pod \"iptables-alerter-hrdzf\" (UID: \"9dd6fe0d-463d-4df8-a826-66922919a981\") " pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.539567 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.539544    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvgf9\" (UniqueName: \"kubernetes.io/projected/441e22c2-c489-4c53-aa0f-4d96208b16ff-kube-api-access-fvgf9\") pod \"ovnkube-node-rm8q8\" (UID: \"441e22c2-c489-4c53-aa0f-4d96208b16ff\") " pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.540193 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.540169    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvf7f\" (UniqueName: \"kubernetes.io/projected/f8ada3ca-4437-4389-908f-3c9d29d19ae8-kube-api-access-zvf7f\") pod \"aws-ebs-csi-driver-node-c6b4c\" (UID: \"f8ada3ca-4437-4389-908f-3c9d29d19ae8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.601825 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.601798    2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:47:39.632299 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632264    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-os-release\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.632299 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632304    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632327    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-cni-multus\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632350    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysconfig\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632368    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-sys\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632405    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgkt6\" (UniqueName: \"kubernetes.io/projected/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-kube-api-access-xgkt6\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632424    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-lib-modules\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632426    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-os-release\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632426    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-cni-multus\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632495    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysconfig\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632505    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-sys\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632489    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-kubernetes\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632443    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-kubernetes\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.632555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632558    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632590    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-system-cni-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632615    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-cni-bin\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632626    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-lib-modules\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632640    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-kubelet\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/937dcb5e-3893-4ff2-a2f6-a3330a232700-kubelet-config\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632699    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-system-cni-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632708    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/937dcb5e-3893-4ff2-a2f6-a3330a232700-dbus\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632734    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-cnibin\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632757    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-os-release\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632783    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-cni-bin\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632784    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq8dw\" (UniqueName: \"kubernetes.io/projected/61f89985-9448-4de8-b773-dbeb59284d44-kube-api-access-wq8dw\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632833    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-cnibin\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632859    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-conf-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-daemon-config\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632923    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-etc-kubernetes\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632951    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f042f86e-21bb-4d7a-9141-e3fce2adadb3-tmp\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632975    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-hosts-file\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.633120 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632987    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-cnibin\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633009    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633036    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-multus-certs\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633041    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-var-lib-kubelet\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633069    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-os-release\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633082    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-multus-certs\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633089    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-cnibin\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633100    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-cni-binary-copy\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633122    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/937dcb5e-3893-4ff2-a2f6-a3330a232700-kubelet-config\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633135    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/937dcb5e-3893-4ff2-a2f6-a3330a232700-dbus\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.632948    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633170    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-hosts-file\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633176    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn2p6\" (UniqueName: \"kubernetes.io/projected/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-kube-api-access-tn2p6\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633116    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-conf-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633203    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-var-lib-kubelet\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633225    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-host\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633250    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-system-cni-dir\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-socket-dir-parent\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.633824 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633296    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-netns\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633321    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-cni-binary-copy\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633343    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-hostroot\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633398    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-modprobe-d\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633421    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysctl-d\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633420    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-etc-kubernetes\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633442    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysctl-conf\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-tuned\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633492    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-netns\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633497    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-hostroot\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633508    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzdrp\" (UniqueName: \"kubernetes.io/projected/f042f86e-21bb-4d7a-9141-e3fce2adadb3-kube-api-access-pzdrp\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-run\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633544    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-socket-dir-parent\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjt5f\" (UniqueName: \"kubernetes.io/projected/7139966e-e862-401e-bf2b-f13f4f334356-kube-api-access-kjt5f\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633585    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/61f89985-9448-4de8-b773-dbeb59284d44-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633586    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-modprobe-d\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633611    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-var-lib-kubelet\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633606    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48938dcf-66f0-4920-be0c-f3735090e6c2-host\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.634565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633636    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-system-cni-dir\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633639    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-host\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633647    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48938dcf-66f0-4920-be0c-f3735090e6c2-serviceca\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633680    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-systemd\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633703    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-tmp-dir\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633748    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-cni-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633807    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633832    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-k8s-cni-cncf-io\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633837    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysctl-d\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633862    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22tfj\" (UniqueName: \"kubernetes.io/projected/48938dcf-66f0-4920-be0c-f3735090e6c2-kube-api-access-22tfj\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633872    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-run\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.633888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.633994    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634003    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48938dcf-66f0-4920-be0c-f3735090e6c2-host\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634010    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/48938dcf-66f0-4920-be0c-f3735090e6c2-serviceca\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.634051    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.634065    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:40.134047377 +0000 UTC m=+3.086914300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:39.635178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634068    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-host-run-k8s-cni-cncf-io\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.635843 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:39.634085    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret podName:937dcb5e-3893-4ff2-a2f6-a3330a232700 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:40.134074149 +0000 UTC m=+3.086941072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret") pod "global-pull-secret-syncer-6hzm9" (UID: "937dcb5e-3893-4ff2-a2f6-a3330a232700") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:39.635843 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634113    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-systemd\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.635843 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634195    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/61f89985-9448-4de8-b773-dbeb59284d44-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.635843 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634263    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-cni-dir\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.635843 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634327    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-sysctl-conf\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.635843 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634369    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-tmp-dir\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.635843 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634562    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-multus-daemon-config\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.635843 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.634569    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-cni-binary-copy\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.636281 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.636252    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f042f86e-21bb-4d7a-9141-e3fce2adadb3-etc-tuned\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.636560 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.636532    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f042f86e-21bb-4d7a-9141-e3fce2adadb3-tmp\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.641191 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.641166    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq8dw\" (UniqueName: \"kubernetes.io/projected/61f89985-9448-4de8-b773-dbeb59284d44-kube-api-access-wq8dw\") pod \"multus-additional-cni-plugins-kmskc\" (UID: \"61f89985-9448-4de8-b773-dbeb59284d44\") " pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.644818 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.644768    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgkt6\" (UniqueName: \"kubernetes.io/projected/a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7-kube-api-access-xgkt6\") pod \"node-resolver-dzgqh\" (UID: \"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7\") " pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.644936 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.644832    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn2p6\" (UniqueName: \"kubernetes.io/projected/ace5f3e3-d61e-41cb-ba3f-e72abf8367b4-kube-api-access-tn2p6\") pod \"multus-cl972\" (UID: \"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4\") " pod="openshift-multus/multus-cl972"
Jun 15 05:47:39.645254 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.645231    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22tfj\" (UniqueName: \"kubernetes.io/projected/48938dcf-66f0-4920-be0c-f3735090e6c2-kube-api-access-22tfj\") pod \"node-ca-4xgl2\" (UID: \"48938dcf-66f0-4920-be0c-f3735090e6c2\") " pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.645538 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.645521    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzdrp\" (UniqueName: \"kubernetes.io/projected/f042f86e-21bb-4d7a-9141-e3fce2adadb3-kube-api-access-pzdrp\") pod \"tuned-7vb9c\" (UID: \"f042f86e-21bb-4d7a-9141-e3fce2adadb3\") " pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.645714 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.645696    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjt5f\" (UniqueName: \"kubernetes.io/projected/7139966e-e862-401e-bf2b-f13f4f334356-kube-api-access-kjt5f\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:39.720207 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.720179    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c"
Jun 15 05:47:39.729148 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.729125    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:47:39.736001 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.735984    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hrdzf"
Jun 15 05:47:39.743671 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.743653    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:47:39.749513 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.749496    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dzgqh"
Jun 15 05:47:39.756065 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.756050    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4xgl2"
Jun 15 05:47:39.762864 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.762838    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kmskc"
Jun 15 05:47:39.769544 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.769523    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7vb9c"
Jun 15 05:47:39.776183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:39.776165    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cl972"
Jun 15 05:47:40.137871 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.137819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:40.138042 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.137893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:40.138042 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.137930    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:40.138042 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:40.138003    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:40.138042 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:40.138028    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:40.138042 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:40.138040    2562 projected.go:196] Error preparing data for projected volume kube-api-access-8z9vj for pod openshift-network-diagnostics/network-check-target-984wm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:40.138042 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:40.138039    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:40.138343 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:40.138039    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:40.138343 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:40.138112    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj podName:c7c76571-ac25-4149-93e0-1c27217e11d1 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:41.138083531 +0000 UTC m=+4.090950455 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8z9vj" (UniqueName: "kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj") pod "network-check-target-984wm" (UID: "c7c76571-ac25-4149-93e0-1c27217e11d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:40.138343 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:40.138134    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret podName:937dcb5e-3893-4ff2-a2f6-a3330a232700 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:41.138123924 +0000 UTC m=+4.090990847 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret") pod "global-pull-secret-syncer-6hzm9" (UID: "937dcb5e-3893-4ff2-a2f6-a3330a232700") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:40.138343 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:40.138151    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:41.138141897 +0000 UTC m=+4.091008821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:40.236358 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:40.236336    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf042f86e_21bb_4d7a_9141_e3fce2adadb3.slice/crio-fb91ae649c25427e5750f76614907cdb24b8f960ae70d765cbb6cc3c67e6d5f2 WatchSource:0}: Error finding container fb91ae649c25427e5750f76614907cdb24b8f960ae70d765cbb6cc3c67e6d5f2: Status 404 returned error can't find the container with id fb91ae649c25427e5750f76614907cdb24b8f960ae70d765cbb6cc3c67e6d5f2
Jun 15 05:47:40.237475 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:40.237456    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ada3ca_4437_4389_908f_3c9d29d19ae8.slice/crio-5f5decc3db525c8daf3a9bdcb25ea7debd208ab2a534831fa7ebd49e6fe759f0 WatchSource:0}: Error finding container 5f5decc3db525c8daf3a9bdcb25ea7debd208ab2a534831fa7ebd49e6fe759f0: Status 404 returned error can't find the container with id 5f5decc3db525c8daf3a9bdcb25ea7debd208ab2a534831fa7ebd49e6fe759f0
Jun 15 05:47:40.239187 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:40.238583    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e05b24_a058_4fef_b061_38cb4453e6c2.slice/crio-805fc7f4071cb57c8daf7bf17bcc56b0eea7491982b7918ecd8c66a72855d0fc WatchSource:0}: Error finding container 805fc7f4071cb57c8daf7bf17bcc56b0eea7491982b7918ecd8c66a72855d0fc: Status 404 returned error can't find the container with id 805fc7f4071cb57c8daf7bf17bcc56b0eea7491982b7918ecd8c66a72855d0fc
Jun 15 05:47:40.240973 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:40.240942    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f89985_9448_4de8_b773_dbeb59284d44.slice/crio-0dcff35eebb91d2f17fbbe1e6e94e4d5189c7a7457dd8792bc8b7130730f8d51 WatchSource:0}: Error finding container 0dcff35eebb91d2f17fbbe1e6e94e4d5189c7a7457dd8792bc8b7130730f8d51: Status 404 returned error can't find the container with id 0dcff35eebb91d2f17fbbe1e6e94e4d5189c7a7457dd8792bc8b7130730f8d51
Jun 15 05:47:40.243979 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:40.243961    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd6fe0d_463d_4df8_a826_66922919a981.slice/crio-35efcf9019cf54a71ecba4788d3894032a581f8fe48ec52ce9f35dd65a682739 WatchSource:0}: Error finding container 35efcf9019cf54a71ecba4788d3894032a581f8fe48ec52ce9f35dd65a682739: Status 404 returned error can't find the container with id 35efcf9019cf54a71ecba4788d3894032a581f8fe48ec52ce9f35dd65a682739
Jun 15 05:47:40.246026 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:40.246003    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48938dcf_66f0_4920_be0c_f3735090e6c2.slice/crio-a7d29ff7057f0d0efca8361bb4adf3816a1fd1d3c734035297900fe1f58d6e3a WatchSource:0}: Error finding container a7d29ff7057f0d0efca8361bb4adf3816a1fd1d3c734035297900fe1f58d6e3a: Status 404 returned error can't find the container with id a7d29ff7057f0d0efca8361bb4adf3816a1fd1d3c734035297900fe1f58d6e3a
Jun 15 05:47:40.247091 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:40.246989    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441e22c2_c489_4c53_aa0f_4d96208b16ff.slice/crio-9df0dda83fb8209a695289681f732def064554f641e87e5f1972e78e228fe57a WatchSource:0}: Error finding container 9df0dda83fb8209a695289681f732def064554f641e87e5f1972e78e228fe57a: Status 404 returned error can't find the container with id 9df0dda83fb8209a695289681f732def064554f641e87e5f1972e78e228fe57a
Jun 15 05:47:40.249466 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:47:40.249361    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a550a5_e9e9_4c12_b69d_8ae4a696ffd7.slice/crio-d2b16a8afa41128ae0f77da0bb2f24ffcca34220c9b15b7fbfcd6f63f263d088 WatchSource:0}: Error finding container d2b16a8afa41128ae0f77da0bb2f24ffcca34220c9b15b7fbfcd6f63f263d088: Status 404 returned error can't find the container with id d2b16a8afa41128ae0f77da0bb2f24ffcca34220c9b15b7fbfcd6f63f263d088
Jun 15 05:47:40.457855 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.457759    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:42:39 +0000 UTC" deadline="2028-01-11 16:32:39.931226042 +0000 UTC"
Jun 15 05:47:40.457855 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.457807    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13810h44m59.473421891s"
Jun 15 05:47:40.491680 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.491654    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzgqh" event={"ID":"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7","Type":"ContainerStarted","Data":"d2b16a8afa41128ae0f77da0bb2f24ffcca34220c9b15b7fbfcd6f63f263d088"}
Jun 15 05:47:40.492689 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.492658    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"9df0dda83fb8209a695289681f732def064554f641e87e5f1972e78e228fe57a"}
Jun 15 05:47:40.493850 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.493823    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hrdzf" event={"ID":"9dd6fe0d-463d-4df8-a826-66922919a981","Type":"ContainerStarted","Data":"35efcf9019cf54a71ecba4788d3894032a581f8fe48ec52ce9f35dd65a682739"}
Jun 15 05:47:40.494909 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.494890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerStarted","Data":"0dcff35eebb91d2f17fbbe1e6e94e4d5189c7a7457dd8792bc8b7130730f8d51"}
Jun 15 05:47:40.495837 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.495819    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gbjb7" event={"ID":"14e05b24-a058-4fef-b061-38cb4453e6c2","Type":"ContainerStarted","Data":"805fc7f4071cb57c8daf7bf17bcc56b0eea7491982b7918ecd8c66a72855d0fc"}
Jun 15 05:47:40.496879 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.496856    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c" event={"ID":"f8ada3ca-4437-4389-908f-3c9d29d19ae8","Type":"ContainerStarted","Data":"5f5decc3db525c8daf3a9bdcb25ea7debd208ab2a534831fa7ebd49e6fe759f0"}
Jun 15 05:47:40.497827 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.497806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vb9c" event={"ID":"f042f86e-21bb-4d7a-9141-e3fce2adadb3","Type":"ContainerStarted","Data":"fb91ae649c25427e5750f76614907cdb24b8f960ae70d765cbb6cc3c67e6d5f2"}
Jun 15 05:47:40.500521 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.500501    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cl972" event={"ID":"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4","Type":"ContainerStarted","Data":"b19dc75c7552ec7c7b17040ad75600fece792a6a517b3d841b24f6328a35be0b"}
Jun 15 05:47:40.501418 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.501400    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xgl2" event={"ID":"48938dcf-66f0-4920-be0c-f3735090e6c2","Type":"ContainerStarted","Data":"a7d29ff7057f0d0efca8361bb4adf3816a1fd1d3c734035297900fe1f58d6e3a"}
Jun 15 05:47:40.502527 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.502505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal" event={"ID":"08d00837dae608698233b66e665078bb","Type":"ContainerStarted","Data":"9807d4bdde4ae47d2e8f0512c9a835517c685b6411e59c943bb68597344d61d1"}
Jun 15 05:47:40.518733 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:40.518692    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-111.ec2.internal" podStartSLOduration=1.518680624 podStartE2EDuration="1.518680624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:40.518543662 +0000 UTC m=+3.471410601" watchObservedRunningTime="2026-06-15 05:47:40.518680624 +0000 UTC m=+3.471547569"
Jun 15 05:47:41.145791 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.145747    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:41.145991 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.145819    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:41.145991 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.145888    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:41.145991 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.145939    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:41.145991 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.145966    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:41.145991 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.145981    2562 projected.go:196] Error preparing data for projected volume kube-api-access-8z9vj for pod openshift-network-diagnostics/network-check-target-984wm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:41.146250 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.146007    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:41.146250 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.146042    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj podName:c7c76571-ac25-4149-93e0-1c27217e11d1 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:43.146024083 +0000 UTC m=+6.098891030 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8z9vj" (UniqueName: "kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj") pod "network-check-target-984wm" (UID: "c7c76571-ac25-4149-93e0-1c27217e11d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:41.146250 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.146062    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:43.146051463 +0000 UTC m=+6.098918392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:41.146250 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.146098    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:41.146250 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.146135    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret podName:937dcb5e-3893-4ff2-a2f6-a3330a232700 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:43.146120294 +0000 UTC m=+6.098987224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret") pod "global-pull-secret-syncer-6hzm9" (UID: "937dcb5e-3893-4ff2-a2f6-a3330a232700") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:41.458582 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.458482    2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:42:39 +0000 UTC" deadline="2027-12-31 21:14:58.513088485 +0000 UTC"
Jun 15 05:47:41.458582 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.458524    2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13551h27m17.054569094s"
Jun 15 05:47:41.486431 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.486373    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:41.486617 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.486539    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:41.486956 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.486937    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:41.487073 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.487053    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:41.487144 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.487114    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:41.487207 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:41.487180    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:41.527056 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.527013    2562 generic.go:356] "Generic (PLEG): container finished" podID="dee957abc27887cd92f63b816a7fda6a" containerID="7fead60e0f63871f94308b1abb2c1f493ba3ec1e6b91c950456b723dd85b57e3" exitCode=0
Jun 15 05:47:41.528027 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:41.527997    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal" event={"ID":"dee957abc27887cd92f63b816a7fda6a","Type":"ContainerDied","Data":"7fead60e0f63871f94308b1abb2c1f493ba3ec1e6b91c950456b723dd85b57e3"}
Jun 15 05:47:42.555098 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:42.554795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal" event={"ID":"dee957abc27887cd92f63b816a7fda6a","Type":"ContainerStarted","Data":"91c77c869d26b798aed6c50d979a81cb21cb5bc195ffc9a6d6d9130be53f1ec5"}
Jun 15 05:47:43.162473 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:43.162357    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:43.162473 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:43.162447    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:43.162473 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:43.162478    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:43.164014 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.163897    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:43.164014 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.163989    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:47.16396836 +0000 UTC m=+10.116835287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:43.167563 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.167542    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:43.167686 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.167618    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret podName:937dcb5e-3893-4ff2-a2f6-a3330a232700 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:47.167599124 +0000 UTC m=+10.120466049 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret") pod "global-pull-secret-syncer-6hzm9" (UID: "937dcb5e-3893-4ff2-a2f6-a3330a232700") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:43.167761 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.167704    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:43.167761 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.167723    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:43.167761 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.167735    2562 projected.go:196] Error preparing data for projected volume kube-api-access-8z9vj for pod openshift-network-diagnostics/network-check-target-984wm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:43.167903 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.167774    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj podName:c7c76571-ac25-4149-93e0-1c27217e11d1 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:47.167763264 +0000 UTC m=+10.120630191 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8z9vj" (UniqueName: "kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj") pod "network-check-target-984wm" (UID: "c7c76571-ac25-4149-93e0-1c27217e11d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:43.485122 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:43.485087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:43.485316 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:43.485087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:43.485316 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:43.485246    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:43.485316 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.485240    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:43.485499 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.485325    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:43.485499 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:43.485405    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:45.484604 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:45.483850    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:45.484604 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:45.483988    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:45.484604 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:45.484358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:45.484604 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:45.484374    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:45.484604 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:45.484479    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:45.484604 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:45.484558    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:47.199421 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:47.199097    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:47.199421 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:47.199154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:47.199421 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:47.199197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:47.199421 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.199346    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:47.199421 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.199361    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:47.199421 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.199370    2562 projected.go:196] Error preparing data for projected volume kube-api-access-8z9vj for pod openshift-network-diagnostics/network-check-target-984wm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:47.199421 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.199435    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj podName:c7c76571-ac25-4149-93e0-1c27217e11d1 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:55.199416523 +0000 UTC m=+18.152283446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8z9vj" (UniqueName: "kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj") pod "network-check-target-984wm" (UID: "c7c76571-ac25-4149-93e0-1c27217e11d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:47.200111 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.199486    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:47.200111 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.199529    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret podName:937dcb5e-3893-4ff2-a2f6-a3330a232700 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:55.199516986 +0000 UTC m=+18.152383909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret") pod "global-pull-secret-syncer-6hzm9" (UID: "937dcb5e-3893-4ff2-a2f6-a3330a232700") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:47.200111 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.199542    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:47.200111 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.199612    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:47:55.199592787 +0000 UTC m=+18.152459728 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:47.484941 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:47.484904    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:47.485095 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.485028    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:47.485414 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:47.485371    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:47.485516 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.485485    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:47.487998 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:47.487976    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:47.488128 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:47.488111    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:49.484038 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:49.484001    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:49.484501 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:49.484150    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:49.484574 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:49.484557    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:49.484678 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:49.484658    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:49.484730 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:49.484713    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:49.484797 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:49.484779    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:51.484397 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:51.484339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:51.484397 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:51.484376    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:51.484860 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:51.484354    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:51.484860 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:51.484516    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:51.484860 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:51.484648    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:51.484860 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:51.484718    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:53.484630 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:53.484543    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:53.484630 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:53.484543    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:53.485138 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:53.484679    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:53.485138 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:53.484696    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:53.485138 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:53.484782    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:53.485138 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:53.484865    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:55.264038 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:55.264001    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:55.264047    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:55.264105    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.264163    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.264207    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.264225    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.264230    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret podName:937dcb5e-3893-4ff2-a2f6-a3330a232700 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:11.264212565 +0000 UTC m=+34.217079489 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret") pod "global-pull-secret-syncer-6hzm9" (UID: "937dcb5e-3893-4ff2-a2f6-a3330a232700") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.264237    2562 projected.go:196] Error preparing data for projected volume kube-api-access-8z9vj for pod openshift-network-diagnostics/network-check-target-984wm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.264236    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.264273    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj podName:c7c76571-ac25-4149-93e0-1c27217e11d1 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:11.264263094 +0000 UTC m=+34.217130016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8z9vj" (UniqueName: "kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj") pod "network-check-target-984wm" (UID: "c7c76571-ac25-4149-93e0-1c27217e11d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:47:55.264528 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.264301    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:11.264279908 +0000 UTC m=+34.217146853 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:47:55.483818 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:55.483781    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:55.483980 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:55.483823    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:55.483980 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:55.483788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:55.483980 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.483911    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:55.484152 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.484017    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:55.484152 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:55.484100    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:57.484651 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:57.484620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:57.485184 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:57.484733    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:57.485256 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:57.485202    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:57.485423 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:57.485306    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:57.485423 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:57.485313    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:57.485669 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:57.485428    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:59.484111 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.483947    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:47:59.484849 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.483959    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:47:59.484849 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:59.484186    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:47:59.484849 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:59.484281    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:47:59.484849 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.484016    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:47:59.484849 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:47:59.484375    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:47:59.590085 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.590048    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dzgqh" event={"ID":"a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7","Type":"ContainerStarted","Data":"33365c41048f73979562be97d9c26c992262e77cc5ed6bdc0e8f599a3fe44b34"}
Jun 15 05:47:59.591985 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.591963    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:47:59.592298 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.592277    2562 generic.go:356] "Generic (PLEG): container finished" podID="441e22c2-c489-4c53-aa0f-4d96208b16ff" containerID="341cf63092529f3852ff88d1aca70226200ddc030d3cd476bb55bda6c164bffa" exitCode=1
Jun 15 05:47:59.592373 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.592349    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerDied","Data":"341cf63092529f3852ff88d1aca70226200ddc030d3cd476bb55bda6c164bffa"}
Jun 15 05:47:59.592475 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.592457    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"ac3576065bc179d7ec8f8d29ec589ce05020a3443187be6a02447b71a016f381"}
Jun 15 05:47:59.593975 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.593949    2562 generic.go:356] "Generic (PLEG): container finished" podID="61f89985-9448-4de8-b773-dbeb59284d44" containerID="3f8b302a50f33f8a6de11b125c320c8691e3ab67f395aa72dd27365e9ac0f7f6" exitCode=0
Jun 15 05:47:59.594071 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.593979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerDied","Data":"3f8b302a50f33f8a6de11b125c320c8691e3ab67f395aa72dd27365e9ac0f7f6"}
Jun 15 05:47:59.595566 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.595521    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gbjb7" event={"ID":"14e05b24-a058-4fef-b061-38cb4453e6c2","Type":"ContainerStarted","Data":"4c733a920393b9a08518265064307e5934061341de6aef8403bd8a2ba415c5e3"}
Jun 15 05:47:59.596949 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.596926    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c" event={"ID":"f8ada3ca-4437-4389-908f-3c9d29d19ae8","Type":"ContainerStarted","Data":"d515ffa36e57af1c12146f19932f09817cd4fd808dd14011673cd947a2d1e502"}
Jun 15 05:47:59.598326 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.598286    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7vb9c" event={"ID":"f042f86e-21bb-4d7a-9141-e3fce2adadb3","Type":"ContainerStarted","Data":"4b0d82c8b84e5d7bccdf48842dde2ff5efe28ff35396bf4e45421274b3864e28"}
Jun 15 05:47:59.599760 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.599737    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cl972" event={"ID":"ace5f3e3-d61e-41cb-ba3f-e72abf8367b4","Type":"ContainerStarted","Data":"0a90f6e02ec52ce81d5356f3bf4c01593e3bb27b97327f33b3a99148e348b917"}
Jun 15 05:47:59.601124 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.601103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4xgl2" event={"ID":"48938dcf-66f0-4920-be0c-f3735090e6c2","Type":"ContainerStarted","Data":"1b0303c13dd74469492755f6eb8f23711f74376855c388f60f55e61ddfe1451c"}
Jun 15 05:47:59.606663 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.606625    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-111.ec2.internal" podStartSLOduration=20.606597183 podStartE2EDuration="20.606597183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:42.574110495 +0000 UTC m=+5.526977441" watchObservedRunningTime="2026-06-15 05:47:59.606597183 +0000 UTC m=+22.559464118"
Jun 15 05:47:59.607270 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.607249    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dzgqh" podStartSLOduration=3.750621321 podStartE2EDuration="22.607241674s" totalImagesPullingTime="18.856620353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:59.606566181 +0000 UTC m=+22.559433125" watchObservedRunningTime="2026-06-15 05:47:59.607241674 +0000 UTC m=+22.560108619"
Jun 15 05:47:59.624844 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.624800    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cl972" podStartSLOduration=3.705028136 podStartE2EDuration="22.624785153s" totalImagesPullingTime="18.919757017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:59.624614975 +0000 UTC m=+22.577481916" watchObservedRunningTime="2026-06-15 05:47:59.624785153 +0000 UTC m=+22.577652097"
Jun 15 05:47:59.641412 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.641344    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gbjb7" podStartSLOduration=8.366004293 podStartE2EDuration="22.641328381s" totalImagesPullingTime="14.275324088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:59.640706668 +0000 UTC m=+22.593573613" watchObservedRunningTime="2026-06-15 05:47:59.641328381 +0000 UTC m=+22.594195363"
Jun 15 05:47:59.676105 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.676061    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4xgl2" podStartSLOduration=3.815804281 podStartE2EDuration="22.676044207s" totalImagesPullingTime="18.860239926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:59.675824055 +0000 UTC m=+22.628691003" watchObservedRunningTime="2026-06-15 05:47:59.676044207 +0000 UTC m=+22.628911232"
Jun 15 05:47:59.694129 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:47:59.694084    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7vb9c" podStartSLOduration=3.77922569 podStartE2EDuration="22.694070194s" totalImagesPullingTime="18.914844504s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:47:59.693718418 +0000 UTC m=+22.646585363" watchObservedRunningTime="2026-06-15 05:47:59.694070194 +0000 UTC m=+22.646937138"
Jun 15 05:48:00.607974 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:00.607794    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:48:00.609412 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:00.608975    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"2bd0d402c263fe80acd0569544975981f5fb0cb76ee90ca86c501e9ecdcca8e4"}
Jun 15 05:48:00.609412 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:00.609095    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"8bbc09e5e97888dc5def8799d82f69a3fe0ddc1a278f47e770ccab0c0e8e7309"}
Jun 15 05:48:00.609412 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:00.609149    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"57e5b0531cebce993eadc2cbd54acd78e7a7058dbbf73444c149bf822abb7d62"}
Jun 15 05:48:00.609412 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:00.609195    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"f44b07dca8ac078f2140d3ec737d4bcd3ce980a1476c7a654765772e1e986c6b"}
Jun 15 05:48:00.711596 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:00.711563    2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 15 05:48:01.484533 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.484493    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:01.484717 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.484493    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:01.484717 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.484495    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:01.484717 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:01.484680    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:48:01.484874 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:01.484707    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:48:01.484874 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:01.484785    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:48:01.488105 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.488020    2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-15T05:48:00.711589014Z","UUID":"d8c64f6a-17f0-4877-93de-b9b89a46b81c","Handler":null,"Name":"","Endpoint":""}
Jun 15 05:48:01.489775 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.489755    2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 15 05:48:01.489898 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.489796    2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 15 05:48:01.612722 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.612688    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hrdzf" event={"ID":"9dd6fe0d-463d-4df8-a826-66922919a981","Type":"ContainerStarted","Data":"b6b0764ba4a0b8dc9993bfe7de6e2dc9c0aacee5fc0e7b835304f3735603ef66"}
Jun 15 05:48:01.615320 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.615287    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c" event={"ID":"f8ada3ca-4437-4389-908f-3c9d29d19ae8","Type":"ContainerStarted","Data":"a0b7403f7972056b316081a029c37f7937107868edda88e784924438fda150c2"}
Jun 15 05:48:01.631006 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:01.630950    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hrdzf" podStartSLOduration=5.728085707 podStartE2EDuration="24.630926281s" totalImagesPullingTime="18.902840574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:01.6306083 +0000 UTC m=+24.583475247" watchObservedRunningTime="2026-06-15 05:48:01.630926281 +0000 UTC m=+24.583793228"
Jun 15 05:48:02.620718 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:02.620671    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c" event={"ID":"f8ada3ca-4437-4389-908f-3c9d29d19ae8","Type":"ContainerStarted","Data":"3ccd5458750fe7226304b0f7b943223a6cda33c7ef9df5a38ca96b9e958831b9"}
Jun 15 05:48:02.624470 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:02.624448    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:48:02.624837 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:02.624806    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"1ca816afadf6e9af69f006b8c90e8c470aaa0ba806a9dd49f081357c40d321a8"}
Jun 15 05:48:03.413250 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:03.413214    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:48:03.414412 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:03.414372    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:48:03.431912 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:03.431858    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c6b4c" podStartSLOduration=5.394876709 podStartE2EDuration="26.431837093s" totalImagesPullingTime="21.036960384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:02.644555418 +0000 UTC m=+25.597422363" watchObservedRunningTime="2026-06-15 05:48:03.431837093 +0000 UTC m=+26.384704039"
Jun 15 05:48:03.484676 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:03.483941    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:03.484676 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:03.483941    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:03.484676 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:03.484070    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:48:03.484676 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:03.484134    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:03.484676 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:03.484234    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:48:03.484676 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:03.484301    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:48:03.627446 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:03.627417    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:48:03.627990 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:03.627890    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gbjb7"
Jun 15 05:48:05.484359 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.484174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:05.484779 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.484174    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:05.484779 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:05.484451    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:48:05.484779 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.484180    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:05.484779 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:05.484512    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:48:05.484779 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:05.484642    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:48:05.635230 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.635204    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:48:05.635576 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.635556    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"527208d94799992196014e68088aa5b75cc18c11c3ccda0a6e5c313b202745b2"}
Jun 15 05:48:05.635817 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.635796    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:48:05.636017 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.636001    2562 scope.go:117] "RemoveContainer" containerID="341cf63092529f3852ff88d1aca70226200ddc030d3cd476bb55bda6c164bffa"
Jun 15 05:48:05.637467 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.637442    2562 generic.go:356] "Generic (PLEG): container finished" podID="61f89985-9448-4de8-b773-dbeb59284d44" containerID="8b3bba3dd8c44aee69dd3acfa11424a9d278eb35826ea00d09c3fd99e5a2ee7c" exitCode=0
Jun 15 05:48:05.637591 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.637523    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerDied","Data":"8b3bba3dd8c44aee69dd3acfa11424a9d278eb35826ea00d09c3fd99e5a2ee7c"}
Jun 15 05:48:05.653544 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:05.653513    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:48:06.643657 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.643495    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:48:06.644360 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.643970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" event={"ID":"441e22c2-c489-4c53-aa0f-4d96208b16ff","Type":"ContainerStarted","Data":"40edd21e224de2c0adf5f613ef0ad003d34ac42d7c2bd7a9d241f2142f440da1"}
Jun 15 05:48:06.644360 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.644306    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:48:06.644512 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.644375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:48:06.646000 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.645977    2562 generic.go:356] "Generic (PLEG): container finished" podID="61f89985-9448-4de8-b773-dbeb59284d44" containerID="4366893247efa14653d7d3fe8308b4349068854c72a069c69217650248609701" exitCode=0
Jun 15 05:48:06.646115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.646020    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerDied","Data":"4366893247efa14653d7d3fe8308b4349068854c72a069c69217650248609701"}
Jun 15 05:48:06.660606 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.660575    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:48:06.732090 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.732037    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8" podStartSLOduration=10.773540187 podStartE2EDuration="29.732021754s" totalImagesPullingTime="18.958481567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:06.684463875 +0000 UTC m=+29.637330819" watchObservedRunningTime="2026-06-15 05:48:06.732021754 +0000 UTC m=+29.684888716"
Jun 15 05:48:06.797634 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.797598    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6hzm9"]
Jun 15 05:48:06.797795 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.797703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:06.797838 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:06.797789    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:48:06.800739 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.800714    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-984wm"]
Jun 15 05:48:06.800897 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.800815    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:06.800946 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:06.800905    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:48:06.801403 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.801356    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cplw2"]
Jun 15 05:48:06.801505 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:06.801466    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:06.801568 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:06.801546    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:48:07.650933 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:07.650836    2562 generic.go:356] "Generic (PLEG): container finished" podID="61f89985-9448-4de8-b773-dbeb59284d44" containerID="703fe3d04d2aee08967a7e9ef981c88cfaaa9e89fc3a83e33b1327a1b4a48c8f" exitCode=0
Jun 15 05:48:07.650933 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:07.650910    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerDied","Data":"703fe3d04d2aee08967a7e9ef981c88cfaaa9e89fc3a83e33b1327a1b4a48c8f"}
Jun 15 05:48:08.484273 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:08.484239    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:08.484497 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:08.484318    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:08.484497 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:08.484358    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:08.484611 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:08.484500    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:48:08.484665 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:08.484646    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:48:08.484734 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:08.484719    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:48:10.484139 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:10.483935    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:10.484669 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:10.483971    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:10.484669 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:10.484232    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-984wm" podUID="c7c76571-ac25-4149-93e0-1c27217e11d1"
Jun 15 05:48:10.484669 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:10.483988    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:10.484669 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:10.484293    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6hzm9" podUID="937dcb5e-3893-4ff2-a2f6-a3330a232700"
Jun 15 05:48:10.484669 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:10.484437    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:48:11.280044 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:11.279998    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:11.280233 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:11.280096    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:11.280233 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:11.280138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:11.280233 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:11.280172    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:11.280233 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:11.280223    2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:11.280367 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:11.280268    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:43.280226656 +0000 UTC m=+66.233093600 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:11.280367 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:11.280284    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret podName:937dcb5e-3893-4ff2-a2f6-a3330a232700 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:43.280277377 +0000 UTC m=+66.233144299 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret") pod "global-pull-secret-syncer-6hzm9" (UID: "937dcb5e-3893-4ff2-a2f6-a3330a232700") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:11.280367 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:11.280304    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:11.280367 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:11.280316    2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:11.280367 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:11.280324    2562 projected.go:196] Error preparing data for projected volume kube-api-access-8z9vj for pod openshift-network-diagnostics/network-check-target-984wm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:11.280367 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:11.280351    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj podName:c7c76571-ac25-4149-93e0-1c27217e11d1 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:43.280340102 +0000 UTC m=+66.233207025 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8z9vj" (UniqueName: "kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj") pod "network-check-target-984wm" (UID: "c7c76571-ac25-4149-93e0-1c27217e11d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:12.284396 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.284348    2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-111.ec2.internal" event="NodeReady"
Jun 15 05:48:12.284787 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.284489    2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jun 15 05:48:12.332595 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.332557    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zwqwd"]
Jun 15 05:48:12.352604 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.352574    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5tnsl"]
Jun 15 05:48:12.352764 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.352698    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.355304 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.355280    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mfr5\""
Jun 15 05:48:12.355483 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.355305    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 15 05:48:12.355483 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.355310    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 15 05:48:12.373105 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.373078    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zwqwd"]
Jun 15 05:48:12.373105 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.373116    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5tnsl"]
Jun 15 05:48:12.373280 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.373199    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5tnsl"
Jun 15 05:48:12.375846 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.375825    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8pttw\""
Jun 15 05:48:12.375977 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.375862    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 15 05:48:12.376138 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.376118    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 15 05:48:12.376242 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.376228    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 15 05:48:12.483995 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.483961    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:12.483995 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.483984    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:12.484255 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.483977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:12.487081 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.487058    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8qp7w\""
Jun 15 05:48:12.487264 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.487057    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 15 05:48:12.487264 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.487101    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 15 05:48:12.487264 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.487117    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s8jdr\""
Jun 15 05:48:12.487264 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.487225    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 15 05:48:12.487450 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.487297    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 15 05:48:12.489152 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.489128    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30915d28-dba8-4a63-a4f6-367fe7a59f4a-config-volume\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.489276 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.489168    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30915d28-dba8-4a63-a4f6-367fe7a59f4a-tmp-dir\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.489276 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.489187    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngl8\" (UniqueName: \"kubernetes.io/projected/f4c07b8d-4964-4899-a3ff-5f9b0fa13728-kube-api-access-6ngl8\") pod \"ingress-canary-5tnsl\" (UID: \"f4c07b8d-4964-4899-a3ff-5f9b0fa13728\") " pod="openshift-ingress-canary/ingress-canary-5tnsl"
Jun 15 05:48:12.489276 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.489243    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4c07b8d-4964-4899-a3ff-5f9b0fa13728-cert\") pod \"ingress-canary-5tnsl\" (UID: \"f4c07b8d-4964-4899-a3ff-5f9b0fa13728\") " pod="openshift-ingress-canary/ingress-canary-5tnsl"
Jun 15 05:48:12.489276 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.489268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w95mm\" (UniqueName: \"kubernetes.io/projected/30915d28-dba8-4a63-a4f6-367fe7a59f4a-kube-api-access-w95mm\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.489467 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.489287    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.590653 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.590610    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30915d28-dba8-4a63-a4f6-367fe7a59f4a-config-volume\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.590653 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.590655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30915d28-dba8-4a63-a4f6-367fe7a59f4a-tmp-dir\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.590882 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.590681    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngl8\" (UniqueName: \"kubernetes.io/projected/f4c07b8d-4964-4899-a3ff-5f9b0fa13728-kube-api-access-6ngl8\") pod \"ingress-canary-5tnsl\" (UID: \"f4c07b8d-4964-4899-a3ff-5f9b0fa13728\") " pod="openshift-ingress-canary/ingress-canary-5tnsl"
Jun 15 05:48:12.590882 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.590825    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4c07b8d-4964-4899-a3ff-5f9b0fa13728-cert\") pod \"ingress-canary-5tnsl\" (UID: \"f4c07b8d-4964-4899-a3ff-5f9b0fa13728\") " pod="openshift-ingress-canary/ingress-canary-5tnsl"
Jun 15 05:48:12.590882 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.590865    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95mm\" (UniqueName: \"kubernetes.io/projected/30915d28-dba8-4a63-a4f6-367fe7a59f4a-kube-api-access-w95mm\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.591024 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.590899    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.591024 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.590958    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30915d28-dba8-4a63-a4f6-367fe7a59f4a-tmp-dir\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.591024 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:12.591009    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:12.591149 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:12.591071    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:48:13.091049701 +0000 UTC m=+36.043916628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:12.591338 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.591314    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/30915d28-dba8-4a63-a4f6-367fe7a59f4a-config-volume\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.595264 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.595245    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4c07b8d-4964-4899-a3ff-5f9b0fa13728-cert\") pod \"ingress-canary-5tnsl\" (UID: \"f4c07b8d-4964-4899-a3ff-5f9b0fa13728\") " pod="openshift-ingress-canary/ingress-canary-5tnsl"
Jun 15 05:48:12.600406 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.600357    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngl8\" (UniqueName: \"kubernetes.io/projected/f4c07b8d-4964-4899-a3ff-5f9b0fa13728-kube-api-access-6ngl8\") pod \"ingress-canary-5tnsl\" (UID: \"f4c07b8d-4964-4899-a3ff-5f9b0fa13728\") " pod="openshift-ingress-canary/ingress-canary-5tnsl"
Jun 15 05:48:12.600743 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.600725    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95mm\" (UniqueName: \"kubernetes.io/projected/30915d28-dba8-4a63-a4f6-367fe7a59f4a-kube-api-access-w95mm\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:12.669907 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.669872    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerStarted","Data":"6d3940a04bbec061d24396c4825e733c68cb4e92ad30a3af9916ad0e00c16e93"}
Jun 15 05:48:12.682481 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.682457    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5tnsl"
Jun 15 05:48:12.882699 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:12.882494    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5tnsl"]
Jun 15 05:48:12.886531 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:48:12.886500    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c07b8d_4964_4899_a3ff_5f9b0fa13728.slice/crio-8fd49619a651b715e91a81b3f9663fa4980b050e28c736c27a78a73a236d51ba WatchSource:0}: Error finding container 8fd49619a651b715e91a81b3f9663fa4980b050e28c736c27a78a73a236d51ba: Status 404 returned error can't find the container with id 8fd49619a651b715e91a81b3f9663fa4980b050e28c736c27a78a73a236d51ba
Jun 15 05:48:13.094638 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:13.094553    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:13.094789 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:13.094714    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:13.094789 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:13.094777    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:48:14.094761979 +0000 UTC m=+37.047628901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:13.674117 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:13.674073    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5tnsl" event={"ID":"f4c07b8d-4964-4899-a3ff-5f9b0fa13728","Type":"ContainerStarted","Data":"8fd49619a651b715e91a81b3f9663fa4980b050e28c736c27a78a73a236d51ba"}
Jun 15 05:48:13.677281 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:13.677251    2562 generic.go:356] "Generic (PLEG): container finished" podID="61f89985-9448-4de8-b773-dbeb59284d44" containerID="6d3940a04bbec061d24396c4825e733c68cb4e92ad30a3af9916ad0e00c16e93" exitCode=0
Jun 15 05:48:13.677453 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:13.677295    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerDied","Data":"6d3940a04bbec061d24396c4825e733c68cb4e92ad30a3af9916ad0e00c16e93"}
Jun 15 05:48:14.102728 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:14.102630    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:14.102886 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:14.102783    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:14.102886 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:14.102848    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:48:16.102833609 +0000 UTC m=+39.055700532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:14.685635 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:14.685598    2562 generic.go:356] "Generic (PLEG): container finished" podID="61f89985-9448-4de8-b773-dbeb59284d44" containerID="92782c73c16fc92b18bf11f1554dc73c30c7df053ca0d93a452f09b787225ce0" exitCode=0
Jun 15 05:48:14.686248 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:14.685669    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerDied","Data":"92782c73c16fc92b18bf11f1554dc73c30c7df053ca0d93a452f09b787225ce0"}
Jun 15 05:48:15.689492 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:15.689447    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5tnsl" event={"ID":"f4c07b8d-4964-4899-a3ff-5f9b0fa13728","Type":"ContainerStarted","Data":"e49b41bcc19843e6fe1edd1a90e6abfe9415fba9c39c22a51e7d7ff768447fb6"}
Jun 15 05:48:15.692483 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:15.692458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kmskc" event={"ID":"61f89985-9448-4de8-b773-dbeb59284d44","Type":"ContainerStarted","Data":"b40b83c04a91fad1f8c330f3974940a030f68d3becf9dd19186bae4a4555042b"}
Jun 15 05:48:15.706329 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:15.706277    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5tnsl" podStartSLOduration=1.590071749 podStartE2EDuration="3.706258926s" totalImagesPullingTime="2.116187177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:15.705878875 +0000 UTC m=+38.658745824" watchObservedRunningTime="2026-06-15 05:48:15.706258926 +0000 UTC m=+38.659125872"
Jun 15 05:48:15.731964 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:15.729522    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kmskc" podStartSLOduration=8.275934656 podStartE2EDuration="38.72950439s" totalImagesPullingTime="30.453569734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:15.727711167 +0000 UTC m=+38.680578112" watchObservedRunningTime="2026-06-15 05:48:15.72950439 +0000 UTC m=+38.682371335"
Jun 15 05:48:16.117543 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:16.117501    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:16.117718 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:16.117611    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:16.117718 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:16.117673    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:48:20.117659343 +0000 UTC m=+43.070526266 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:20.144813 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:20.144570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:20.145204 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:20.144734    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:20.145204 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:20.144951    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:48:28.144932536 +0000 UTC m=+51.097799466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:28.199345 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:28.199305    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:28.199789 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:28.199503    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:28.199789 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:28.199579    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:48:44.199560501 +0000 UTC m=+67.152427426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:38.664631 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:38.664601    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rm8q8"
Jun 15 05:48:43.300666 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.300619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:43.300666 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.300671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:48:43.301145 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.300723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:43.303752 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.303727    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 15 05:48:43.303837 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.303729    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 15 05:48:43.303837 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.303790    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 15 05:48:43.311225 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:43.311200    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 15 05:48:43.311310 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:43.311255    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:47.311240521 +0000 UTC m=+130.264107444 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : secret "metrics-daemon-secret" not found
Jun 15 05:48:43.313731 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.313717    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 15 05:48:43.314726 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.314710    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/937dcb5e-3893-4ff2-a2f6-a3330a232700-original-pull-secret\") pod \"global-pull-secret-syncer-6hzm9\" (UID: \"937dcb5e-3893-4ff2-a2f6-a3330a232700\") " pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:43.324004 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.323976    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9vj\" (UniqueName: \"kubernetes.io/projected/c7c76571-ac25-4149-93e0-1c27217e11d1-kube-api-access-8z9vj\") pod \"network-check-target-984wm\" (UID: \"c7c76571-ac25-4149-93e0-1c27217e11d1\") " pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:43.404124 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.404096    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s8jdr\""
Jun 15 05:48:43.411602 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.411583    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:43.413860 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.413841    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6hzm9"
Jun 15 05:48:43.577716 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.577634    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-984wm"]
Jun 15 05:48:43.581240 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:48:43.581214    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c76571_ac25_4149_93e0_1c27217e11d1.slice/crio-5015634f9ee6d22168d427ba43b4f40357b245df12d1397511ac5384fbc94591 WatchSource:0}: Error finding container 5015634f9ee6d22168d427ba43b4f40357b245df12d1397511ac5384fbc94591: Status 404 returned error can't find the container with id 5015634f9ee6d22168d427ba43b4f40357b245df12d1397511ac5384fbc94591
Jun 15 05:48:43.593586 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.593563    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6hzm9"]
Jun 15 05:48:43.596216 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:48:43.596192    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod937dcb5e_3893_4ff2_a2f6_a3330a232700.slice/crio-ad6c8de54c5fd742def8baff27ea95734f06b82444a3432679d877d30efd8175 WatchSource:0}: Error finding container ad6c8de54c5fd742def8baff27ea95734f06b82444a3432679d877d30efd8175: Status 404 returned error can't find the container with id ad6c8de54c5fd742def8baff27ea95734f06b82444a3432679d877d30efd8175
Jun 15 05:48:43.757781 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.757749    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6hzm9" event={"ID":"937dcb5e-3893-4ff2-a2f6-a3330a232700","Type":"ContainerStarted","Data":"ad6c8de54c5fd742def8baff27ea95734f06b82444a3432679d877d30efd8175"}
Jun 15 05:48:43.758817 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:43.758793    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-984wm" event={"ID":"c7c76571-ac25-4149-93e0-1c27217e11d1","Type":"ContainerStarted","Data":"5015634f9ee6d22168d427ba43b4f40357b245df12d1397511ac5384fbc94591"}
Jun 15 05:48:44.205819 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:44.205777    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:48:44.205992 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:44.205930    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:48:44.206043 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:48:44.206002    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:49:16.205985549 +0000 UTC m=+99.158852483 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:48:48.774812 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:48.774774    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6hzm9" event={"ID":"937dcb5e-3893-4ff2-a2f6-a3330a232700","Type":"ContainerStarted","Data":"420d874594979de7898846727bf75c194239d7c1c1e105d4dab5560b3f5dd533"}
Jun 15 05:48:48.776267 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:48.776242    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-984wm" event={"ID":"c7c76571-ac25-4149-93e0-1c27217e11d1","Type":"ContainerStarted","Data":"477a3388d5b613c50ee4c005a18ff21a28b0375c3aaf83817cc30c05e5ff200b"}
Jun 15 05:48:48.776415 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:48.776375    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:48:48.795916 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:48.795862    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6hzm9" podStartSLOduration=67.31418664 podStartE2EDuration="1m11.795847875s" totalImagesPullingTime="4.481661235s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:48.795195766 +0000 UTC m=+71.748062725" watchObservedRunningTime="2026-06-15 05:48:48.795847875 +0000 UTC m=+71.748714798"
Jun 15 05:48:48.814439 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:48.814359    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-984wm" podStartSLOduration=67.320509087 podStartE2EDuration="1m11.814342543s" totalImagesPullingTime="4.493833456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:48.813667484 +0000 UTC m=+71.766534429" watchObservedRunningTime="2026-06-15 05:48:48.814342543 +0000 UTC m=+71.767209488"
Jun 15 05:48:56.451722 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.451598    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"]
Jun 15 05:48:56.453289 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.453274    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.457002 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.456975    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jun 15 05:48:56.457124 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.457049    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jun 15 05:48:56.457775 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.457760    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jun 15 05:48:56.457834 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.457766    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jun 15 05:48:56.469578 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.469557    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"]
Jun 15 05:48:56.489285 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.489255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-klusterlet-config\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.489455 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.489307    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6mw\" (UniqueName: \"kubernetes.io/projected/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-kube-api-access-9b6mw\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.489455 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.489336    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-tmp\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.589903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.589864    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6mw\" (UniqueName: \"kubernetes.io/projected/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-kube-api-access-9b6mw\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.589903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.589909    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-tmp\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.590146 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.589946    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-klusterlet-config\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.590337 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.590313    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-tmp\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.592449 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.592432    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-klusterlet-config\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.598876 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.598846    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6mw\" (UniqueName: \"kubernetes.io/projected/35d048a8-5e0a-4874-91f5-7c9cfcaa7416-kube-api-access-9b6mw\") pod \"klusterlet-addon-workmgr-657b494b7b-jhfnd\" (UID: \"35d048a8-5e0a-4874-91f5-7c9cfcaa7416\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.762203 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.762170    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:48:56.904019 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:56.901725    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"]
Jun 15 05:48:56.907486 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:48:56.907457    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d048a8_5e0a_4874_91f5_7c9cfcaa7416.slice/crio-571200a663a2f47aa79aa210737d0eb0697bbe2bc464ab0121eed6394712559f WatchSource:0}: Error finding container 571200a663a2f47aa79aa210737d0eb0697bbe2bc464ab0121eed6394712559f: Status 404 returned error can't find the container with id 571200a663a2f47aa79aa210737d0eb0697bbe2bc464ab0121eed6394712559f
Jun 15 05:48:57.800479 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:48:57.800421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd" event={"ID":"35d048a8-5e0a-4874-91f5-7c9cfcaa7416","Type":"ContainerStarted","Data":"571200a663a2f47aa79aa210737d0eb0697bbe2bc464ab0121eed6394712559f"}
Jun 15 05:49:00.809658 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:49:00.809570    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd" event={"ID":"35d048a8-5e0a-4874-91f5-7c9cfcaa7416","Type":"ContainerStarted","Data":"780b2350bab57c6567dff0bebb67ae10f9aa59b73af5a3ca6788d912b27a37f7"}
Jun 15 05:49:00.810067 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:49:00.809787    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:49:00.811411 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:49:00.811374    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:49:00.829823 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:49:00.829776    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd" podStartSLOduration=1.18246983 podStartE2EDuration="4.829763725s" totalImagesPullingTime="3.647293895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:00.829331484 +0000 UTC m=+83.782198429" watchObservedRunningTime="2026-06-15 05:49:00.829763725 +0000 UTC m=+83.782630669"
Jun 15 05:49:16.227307 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:49:16.227261    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:49:16.227800 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:49:16.227438    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:49:16.227800 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:49:16.227518    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:50:20.227497965 +0000 UTC m=+163.180364890 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:49:19.780744 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:49:19.780714    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-984wm"
Jun 15 05:49:47.335681 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:49:47.335626    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:49:47.336248 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:49:47.335780    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 15 05:49:47.336248 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:49:47.335856    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:49.335834619 +0000 UTC m=+252.288701548 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : secret "metrics-daemon-secret" not found
Jun 15 05:50:15.362551 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:50:15.362509    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zwqwd" podUID="30915d28-dba8-4a63-a4f6-367fe7a59f4a"
Jun 15 05:50:15.495299 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:50:15.495254    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:50:15.991142 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:50:15.991113    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:50:20.265971 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:50:20.265931    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:50:20.266424 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:50:20.266075    2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jun 15 05:50:20.266424 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:50:20.266153    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls podName:30915d28-dba8-4a63-a4f6-367fe7a59f4a nodeName:}" failed. No retries permitted until 2026-06-15 05:52:22.266134033 +0000 UTC m=+285.219000956 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls") pod "dns-default-zwqwd" (UID: "30915d28-dba8-4a63-a4f6-367fe7a59f4a") : secret "dns-default-metrics-tls" not found
Jun 15 05:50:29.484111 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:50:29.484072    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:51:00.810298 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:00.810241    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd" podUID="35d048a8-5e0a-4874-91f5-7c9cfcaa7416" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused"
Jun 15 05:51:01.106109 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:01.106075    2562 generic.go:356] "Generic (PLEG): container finished" podID="35d048a8-5e0a-4874-91f5-7c9cfcaa7416" containerID="780b2350bab57c6567dff0bebb67ae10f9aa59b73af5a3ca6788d912b27a37f7" exitCode=1
Jun 15 05:51:01.106297 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:01.106138    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd" event={"ID":"35d048a8-5e0a-4874-91f5-7c9cfcaa7416","Type":"ContainerDied","Data":"780b2350bab57c6567dff0bebb67ae10f9aa59b73af5a3ca6788d912b27a37f7"}
Jun 15 05:51:01.106541 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:01.106524    2562 scope.go:117] "RemoveContainer" containerID="780b2350bab57c6567dff0bebb67ae10f9aa59b73af5a3ca6788d912b27a37f7"
Jun 15 05:51:02.110853 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:02.110812    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd" event={"ID":"35d048a8-5e0a-4874-91f5-7c9cfcaa7416","Type":"ContainerStarted","Data":"4716f8b3992430d34e134bf84327a25ed82832401e12691b0b07c4451e38c4ab"}
Jun 15 05:51:02.111232 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:02.111101    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:51:02.111678 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:02.111662    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-657b494b7b-jhfnd"
Jun 15 05:51:32.002042 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.002007    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b"]
Jun 15 05:51:32.004960 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.004937    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"]
Jun 15 05:51:32.005096 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.005053    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b"
Jun 15 05:51:32.007575 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.007556    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.009163 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.009145    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5xctv\""
Jun 15 05:51:32.010137 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.010117    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jun 15 05:51:32.010137 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.010128    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 15 05:51:32.010312 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.010144    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 15 05:51:32.012137 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.012116    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-27ww5\""
Jun 15 05:51:32.012228 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.012159    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jun 15 05:51:32.013318 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.013292    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b"]
Jun 15 05:51:32.020427 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.020402    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"]
Jun 15 05:51:32.051985 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.051946    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.051985 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.051984    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r66hn\" (UniqueName: \"kubernetes.io/projected/0bde2cab-f15d-4308-b2e7-5c02547676a3-kube-api-access-r66hn\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.052216 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.052008    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0bde2cab-f15d-4308-b2e7-5c02547676a3-telemetry-config\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.052216 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.052058    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnv8n\" (UniqueName: \"kubernetes.io/projected/8ee9b7bb-801c-4c8f-83d0-82872c001eac-kube-api-access-nnv8n\") pod \"network-check-source-6947749cfc-6cq7b\" (UID: \"8ee9b7bb-801c-4c8f-83d0-82872c001eac\") " pod="openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b"
Jun 15 05:51:32.092762 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.092727    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb"]
Jun 15 05:51:32.095746 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.095728    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb"
Jun 15 05:51:32.099215 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.099054    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jun 15 05:51:32.099498 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.099478    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:51:32.099595 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.099507    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-zqq8m\""
Jun 15 05:51:32.100099 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.100081    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-786c49c76b-6n788"]
Jun 15 05:51:32.102712 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.102694    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.105749 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.105729    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jun 15 05:51:32.106696 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.106367    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g2hj5\""
Jun 15 05:51:32.106696 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.106565    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jun 15 05:51:32.106696 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.106617    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jun 15 05:51:32.111565 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.110983    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb"]
Jun 15 05:51:32.114909 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.114887    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jun 15 05:51:32.134838 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.134806    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-786c49c76b-6n788"]
Jun 15 05:51:32.153297 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153268    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-certificates\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.153297 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-trusted-ca\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.153519 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153325    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.153519 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r66hn\" (UniqueName: \"kubernetes.io/projected/0bde2cab-f15d-4308-b2e7-5c02547676a3-kube-api-access-r66hn\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.153519 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153375    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-image-registry-private-configuration\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.153519 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgfp8\" (UniqueName: \"kubernetes.io/projected/7e8cd637-4880-4ab6-9585-d58abfef84c1-kube-api-access-wgfp8\") pod \"volume-data-source-validator-bd4ccd799-nbhgb\" (UID: \"7e8cd637-4880-4ab6-9585-d58abfef84c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb"
Jun 15 05:51:32.153519 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0bde2cab-f15d-4308-b2e7-5c02547676a3-telemetry-config\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.153519 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.153468    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:32.153519 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153474    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-ca-trust-extracted\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.153519 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153502    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-installation-pull-secrets\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.153869 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.153536    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls podName:0bde2cab-f15d-4308-b2e7-5c02547676a3 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:32.653513237 +0000 UTC m=+235.606380161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-f9d7df769-d45f6" (UID: "0bde2cab-f15d-4308-b2e7-5c02547676a3") : secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:32.153869 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153551    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-bound-sa-token\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.153869 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153570    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnr5\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-kube-api-access-tdnr5\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.153869 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnv8n\" (UniqueName: \"kubernetes.io/projected/8ee9b7bb-801c-4c8f-83d0-82872c001eac-kube-api-access-nnv8n\") pod \"network-check-source-6947749cfc-6cq7b\" (UID: \"8ee9b7bb-801c-4c8f-83d0-82872c001eac\") " pod="openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b"
Jun 15 05:51:32.153869 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.153666    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.154130 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.154113    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0bde2cab-f15d-4308-b2e7-5c02547676a3-telemetry-config\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.166965 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.166933    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnv8n\" (UniqueName: \"kubernetes.io/projected/8ee9b7bb-801c-4c8f-83d0-82872c001eac-kube-api-access-nnv8n\") pod \"network-check-source-6947749cfc-6cq7b\" (UID: \"8ee9b7bb-801c-4c8f-83d0-82872c001eac\") " pod="openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b"
Jun 15 05:51:32.167246 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.167229    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r66hn\" (UniqueName: \"kubernetes.io/projected/0bde2cab-f15d-4308-b2e7-5c02547676a3-kube-api-access-r66hn\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.254569 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.254453    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.254569 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.254519    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-certificates\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.254569 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.254536    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-trusted-ca\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.254569 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.254570    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-image-registry-private-configuration\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.254907 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.254588    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgfp8\" (UniqueName: \"kubernetes.io/projected/7e8cd637-4880-4ab6-9585-d58abfef84c1-kube-api-access-wgfp8\") pod \"volume-data-source-validator-bd4ccd799-nbhgb\" (UID: \"7e8cd637-4880-4ab6-9585-d58abfef84c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb"
Jun 15 05:51:32.254907 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.254617    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-ca-trust-extracted\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.254907 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.254621    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:51:32.254907 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.254648    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786c49c76b-6n788: secret "image-registry-tls" not found
Jun 15 05:51:32.254907 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.254731    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls podName:ba1dd848-9482-467a-aefc-0c4dd0da0f5b nodeName:}" failed. No retries permitted until 2026-06-15 05:51:32.754708806 +0000 UTC m=+235.707575728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls") pod "image-registry-786c49c76b-6n788" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b") : secret "image-registry-tls" not found
Jun 15 05:51:32.255302 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.255082    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-ca-trust-extracted\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.255302 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.254645    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-installation-pull-secrets\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.255302 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.255166    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-bound-sa-token\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.255302 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.255200    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnr5\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-kube-api-access-tdnr5\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.255302 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.255241    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-certificates\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.255748 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.255724    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-trusted-ca\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.257361 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.257335    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-image-registry-private-configuration\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.257476 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.257338    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-installation-pull-secrets\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.269466 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.269433    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnr5\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-kube-api-access-tdnr5\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.269763 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.269739    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgfp8\" (UniqueName: \"kubernetes.io/projected/7e8cd637-4880-4ab6-9585-d58abfef84c1-kube-api-access-wgfp8\") pod \"volume-data-source-validator-bd4ccd799-nbhgb\" (UID: \"7e8cd637-4880-4ab6-9585-d58abfef84c1\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb"
Jun 15 05:51:32.270146 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.270120    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-bound-sa-token\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.316114 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.316070    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b"
Jun 15 05:51:32.404829 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.404792    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb"
Jun 15 05:51:32.442656 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.442615    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b"]
Jun 15 05:51:32.445915 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:51:32.445890    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee9b7bb_801c_4c8f_83d0_82872c001eac.slice/crio-36fdf2d275fb089da6b5ca192295312575f4b0d97c367f34d72bffb770bcbcf1 WatchSource:0}: Error finding container 36fdf2d275fb089da6b5ca192295312575f4b0d97c367f34d72bffb770bcbcf1: Status 404 returned error can't find the container with id 36fdf2d275fb089da6b5ca192295312575f4b0d97c367f34d72bffb770bcbcf1
Jun 15 05:51:32.536929 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.536898    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb"]
Jun 15 05:51:32.541333 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:51:32.541304    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8cd637_4880_4ab6_9585_d58abfef84c1.slice/crio-3d94feaad05571a443f445d966869b9245415a609f2529fa1b0e30dfb8ef7501 WatchSource:0}: Error finding container 3d94feaad05571a443f445d966869b9245415a609f2529fa1b0e30dfb8ef7501: Status 404 returned error can't find the container with id 3d94feaad05571a443f445d966869b9245415a609f2529fa1b0e30dfb8ef7501
Jun 15 05:51:32.659176 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.659139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:32.659413 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.659289    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:32.659413 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.659354    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls podName:0bde2cab-f15d-4308-b2e7-5c02547676a3 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:33.659339563 +0000 UTC m=+236.612206486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-f9d7df769-d45f6" (UID: "0bde2cab-f15d-4308-b2e7-5c02547676a3") : secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:32.759803 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:32.759713    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:32.759969 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.759870    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:51:32.759969 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.759889    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786c49c76b-6n788: secret "image-registry-tls" not found
Jun 15 05:51:32.759969 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:32.759943    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls podName:ba1dd848-9482-467a-aefc-0c4dd0da0f5b nodeName:}" failed. No retries permitted until 2026-06-15 05:51:33.759929159 +0000 UTC m=+236.712796083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls") pod "image-registry-786c49c76b-6n788" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b") : secret "image-registry-tls" not found
Jun 15 05:51:33.208497 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:33.208406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b" event={"ID":"8ee9b7bb-801c-4c8f-83d0-82872c001eac","Type":"ContainerStarted","Data":"6bb653a14c537f0d8161fd1cf7edf3fd9ed80cd8848f850b4be200d7f4042a40"}
Jun 15 05:51:33.208497 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:33.208470    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b" event={"ID":"8ee9b7bb-801c-4c8f-83d0-82872c001eac","Type":"ContainerStarted","Data":"36fdf2d275fb089da6b5ca192295312575f4b0d97c367f34d72bffb770bcbcf1"}
Jun 15 05:51:33.209675 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:33.209639    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb" event={"ID":"7e8cd637-4880-4ab6-9585-d58abfef84c1","Type":"ContainerStarted","Data":"3d94feaad05571a443f445d966869b9245415a609f2529fa1b0e30dfb8ef7501"}
Jun 15 05:51:33.227267 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:33.227210    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6947749cfc-6cq7b" podStartSLOduration=2.227194123 podStartE2EDuration="2.227194123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:51:33.226353549 +0000 UTC m=+236.179220494" watchObservedRunningTime="2026-06-15 05:51:33.227194123 +0000 UTC m=+236.180061068"
Jun 15 05:51:33.666225 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:33.666197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:33.666392 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:33.666340    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:33.666456 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:33.666412    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls podName:0bde2cab-f15d-4308-b2e7-5c02547676a3 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:35.666395182 +0000 UTC m=+238.619262107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-f9d7df769-d45f6" (UID: "0bde2cab-f15d-4308-b2e7-5c02547676a3") : secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:33.767241 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:33.767210    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:33.767413 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:33.767324    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:51:33.767413 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:33.767338    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786c49c76b-6n788: secret "image-registry-tls" not found
Jun 15 05:51:33.767501 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:33.767415    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls podName:ba1dd848-9482-467a-aefc-0c4dd0da0f5b nodeName:}" failed. No retries permitted until 2026-06-15 05:51:35.767373246 +0000 UTC m=+238.720240176 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls") pod "image-registry-786c49c76b-6n788" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b") : secret "image-registry-tls" not found
Jun 15 05:51:34.213493 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:34.213398    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb" event={"ID":"7e8cd637-4880-4ab6-9585-d58abfef84c1","Type":"ContainerStarted","Data":"311f70d67b1a22baf893a4034fcdcc517b089af283c1f97fae062b2be5f587f6"}
Jun 15 05:51:34.231442 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:34.231371    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-bd4ccd799-nbhgb" podStartSLOduration=1.3792257989999999 podStartE2EDuration="2.231356471s" totalImagesPullingTime="852.130672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:51:34.230759136 +0000 UTC m=+237.183626168" watchObservedRunningTime="2026-06-15 05:51:34.231356471 +0000 UTC m=+237.184223417"
Jun 15 05:51:35.682071 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:35.682040    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:35.682495 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:35.682461    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:35.682593 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:35.682580    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls podName:0bde2cab-f15d-4308-b2e7-5c02547676a3 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:39.682557568 +0000 UTC m=+242.635424502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-f9d7df769-d45f6" (UID: "0bde2cab-f15d-4308-b2e7-5c02547676a3") : secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:35.783448 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:35.783415    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:35.783610 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:35.783536    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:51:35.783610 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:35.783548    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786c49c76b-6n788: secret "image-registry-tls" not found
Jun 15 05:51:35.783610 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:35.783603    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls podName:ba1dd848-9482-467a-aefc-0c4dd0da0f5b nodeName:}" failed. No retries permitted until 2026-06-15 05:51:39.783589878 +0000 UTC m=+242.736456801 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls") pod "image-registry-786c49c76b-6n788" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b") : secret "image-registry-tls" not found
Jun 15 05:51:37.393508 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.393474    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm"]
Jun 15 05:51:37.396631 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.396608    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm"
Jun 15 05:51:37.399180 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.399157    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jun 15 05:51:37.400278 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.400260    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-t676h\""
Jun 15 05:51:37.400350 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.400292    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jun 15 05:51:37.406209 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.406189    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm"]
Jun 15 05:51:37.497242 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.497215    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnhd9\" (UniqueName: \"kubernetes.io/projected/9bab946d-a7ea-473e-9e9e-00389436e017-kube-api-access-hnhd9\") pod \"migrator-d4bcd4867-9wjqm\" (UID: \"9bab946d-a7ea-473e-9e9e-00389436e017\") " pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm"
Jun 15 05:51:37.598563 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.598526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnhd9\" (UniqueName: \"kubernetes.io/projected/9bab946d-a7ea-473e-9e9e-00389436e017-kube-api-access-hnhd9\") pod \"migrator-d4bcd4867-9wjqm\" (UID: \"9bab946d-a7ea-473e-9e9e-00389436e017\") " pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm"
Jun 15 05:51:37.608822 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.608796    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnhd9\" (UniqueName: \"kubernetes.io/projected/9bab946d-a7ea-473e-9e9e-00389436e017-kube-api-access-hnhd9\") pod \"migrator-d4bcd4867-9wjqm\" (UID: \"9bab946d-a7ea-473e-9e9e-00389436e017\") " pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm"
Jun 15 05:51:37.705044 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.704965    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm"
Jun 15 05:51:37.782331 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.782300    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dvd67"]
Jun 15 05:51:37.785731 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.785706    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:37.789396 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.789358    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 15 05:51:37.789903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.789873    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 15 05:51:37.790281 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.790261    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-snc9v\""
Jun 15 05:51:37.791026 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.791009    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 15 05:51:37.791128 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.791109    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 15 05:51:37.807774 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.807747    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dvd67"]
Jun 15 05:51:37.837553 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.837518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm"]
Jun 15 05:51:37.840466 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:51:37.840439    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bab946d_a7ea_473e_9e9e_00389436e017.slice/crio-6f85b0e8f3444900a3429becb52f5ffb8231035d27d058ad4ee00e2f5a9b54ba WatchSource:0}: Error finding container 6f85b0e8f3444900a3429becb52f5ffb8231035d27d058ad4ee00e2f5a9b54ba: Status 404 returned error can't find the container with id 6f85b0e8f3444900a3429becb52f5ffb8231035d27d058ad4ee00e2f5a9b54ba
Jun 15 05:51:37.901062 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.901016    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbbm2\" (UniqueName: \"kubernetes.io/projected/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-api-access-tbbm2\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:37.901268 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.901133    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5861fb58-e417-4d6d-8720-81f66fbca9d4-crio-socket\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:37.901268 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.901178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:37.901268 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.901217    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:37.901268 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:37.901246    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5861fb58-e417-4d6d-8720-81f66fbca9d4-data-volume\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.002490 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.002462    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5861fb58-e417-4d6d-8720-81f66fbca9d4-crio-socket\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.002661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.002498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.002661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.002522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.002661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.002544    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5861fb58-e417-4d6d-8720-81f66fbca9d4-data-volume\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.002661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.002605    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbbm2\" (UniqueName: \"kubernetes.io/projected/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-api-access-tbbm2\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.002661 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:38.002627    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:38.002661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.002602    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5861fb58-e417-4d6d-8720-81f66fbca9d4-crio-socket\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.003015 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:38.002631    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:38.003015 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:38.002691    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:38.502675805 +0000 UTC m=+241.455542729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:38.003015 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:38.002727    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:38.502715386 +0000 UTC m=+241.455582309 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:38.003015 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.002923    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5861fb58-e417-4d6d-8720-81f66fbca9d4-data-volume\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.011647 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.011626    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbbm2\" (UniqueName: \"kubernetes.io/projected/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-api-access-tbbm2\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.224868 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.224831    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm" event={"ID":"9bab946d-a7ea-473e-9e9e-00389436e017","Type":"ContainerStarted","Data":"6f85b0e8f3444900a3429becb52f5ffb8231035d27d058ad4ee00e2f5a9b54ba"}
Jun 15 05:51:38.511165 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.511127    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.511644 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:38.511184    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:38.511644 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:38.511328    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:38.511644 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:38.511330    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:38.511644 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:38.511444    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:39.511421491 +0000 UTC m=+242.464288420 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:38.511644 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:38.511588    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:39.511535423 +0000 UTC m=+242.464402356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:39.189961 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.189879    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-tl67r"]
Jun 15 05:51:39.192780 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.192765    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.195589 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.195562    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jun 15 05:51:39.196512 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.196488    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jun 15 05:51:39.196512 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.196502    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jun 15 05:51:39.196682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.196501    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jun 15 05:51:39.196682 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.196501    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7tnn8\""
Jun 15 05:51:39.204882 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.204862    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-tl67r"]
Jun 15 05:51:39.231274 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.231238    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm" event={"ID":"9bab946d-a7ea-473e-9e9e-00389436e017","Type":"ContainerStarted","Data":"d172521dd379bdbb0e112dd61210f7200a43a64e5e8b35ca322b1a9461ea1389"}
Jun 15 05:51:39.231449 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.231292    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm" event={"ID":"9bab946d-a7ea-473e-9e9e-00389436e017","Type":"ContainerStarted","Data":"02f072e95b7de22c40ace1c51cdc7bac5caf9e32c669ca473d282f90a3f126b3"}
Jun 15 05:51:39.249031 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.248989    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d4bcd4867-9wjqm" podStartSLOduration=1.205933533 podStartE2EDuration="2.248974538s" totalImagesPullingTime="1.043041005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:51:39.248204479 +0000 UTC m=+242.201071436" watchObservedRunningTime="2026-06-15 05:51:39.248974538 +0000 UTC m=+242.201841482"
Jun 15 05:51:39.318641 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.318600    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-signing-cabundle\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.318810 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.318671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t2qd\" (UniqueName: \"kubernetes.io/projected/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-kube-api-access-6t2qd\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.318810 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.318709    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-signing-key\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.419539 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.419498    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-signing-key\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.419662 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.419565    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-signing-cabundle\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.419662 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.419600    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t2qd\" (UniqueName: \"kubernetes.io/projected/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-kube-api-access-6t2qd\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.420282 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.420260    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-signing-cabundle\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.422213 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.422192    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-signing-key\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.430214 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.430185    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t2qd\" (UniqueName: \"kubernetes.io/projected/0d519d85-bcda-4a59-a2e2-92e6e0e16dae-kube-api-access-6t2qd\") pod \"service-ca-7b994bc59-tl67r\" (UID: \"0d519d85-bcda-4a59-a2e2-92e6e0e16dae\") " pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.501466 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.501440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-7b994bc59-tl67r"
Jun 15 05:51:39.520522 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.520479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:39.520898 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.520543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:39.520898 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.520641    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:39.520898 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.520690    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:39.520898 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.520703    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:41.520688544 +0000 UTC m=+244.473555472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:39.520898 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.520774    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:41.520754969 +0000 UTC m=+244.473621896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:39.627000 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.626969    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-7b994bc59-tl67r"]
Jun 15 05:51:39.629742 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:51:39.629716    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d519d85_bcda_4a59_a2e2_92e6e0e16dae.slice/crio-f97f66886aa1baf6786532a84b9de704e8fb7ac0a3603d09d9d36453e5cd913b WatchSource:0}: Error finding container f97f66886aa1baf6786532a84b9de704e8fb7ac0a3603d09d9d36453e5cd913b: Status 404 returned error can't find the container with id f97f66886aa1baf6786532a84b9de704e8fb7ac0a3603d09d9d36453e5cd913b
Jun 15 05:51:39.722196 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.722154    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:39.722411 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.722305    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:39.722411 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.722373    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls podName:0bde2cab-f15d-4308-b2e7-5c02547676a3 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:47.72235791 +0000 UTC m=+250.675224833 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-f9d7df769-d45f6" (UID: "0bde2cab-f15d-4308-b2e7-5c02547676a3") : secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:39.823185 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:39.823089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:39.823317 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.823217    2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jun 15 05:51:39.823317 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.823240    2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-786c49c76b-6n788: secret "image-registry-tls" not found
Jun 15 05:51:39.823317 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:39.823293    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls podName:ba1dd848-9482-467a-aefc-0c4dd0da0f5b nodeName:}" failed. No retries permitted until 2026-06-15 05:51:47.823277601 +0000 UTC m=+250.776144523 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls") pod "image-registry-786c49c76b-6n788" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b") : secret "image-registry-tls" not found
Jun 15 05:51:40.236351 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:40.236310    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7b994bc59-tl67r" event={"ID":"0d519d85-bcda-4a59-a2e2-92e6e0e16dae","Type":"ContainerStarted","Data":"f97f66886aa1baf6786532a84b9de704e8fb7ac0a3603d09d9d36453e5cd913b"}
Jun 15 05:51:40.367627 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:40.367597    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzgqh_a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7/dns-node-resolver/0.log"
Jun 15 05:51:41.367974 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:41.367948    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xgl2_48938dcf-66f0-4920-be0c-f3735090e6c2/node-ca/0.log"
Jun 15 05:51:41.539124 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:41.539089    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:41.539124 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:41.539143    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:41.539415 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:41.539228    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:41.539415 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:41.539267    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:41.539415 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:41.539296    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:45.539281373 +0000 UTC m=+248.492148296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:41.539415 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:41.539316    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:45.539301895 +0000 UTC m=+248.492168821 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:42.167562 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:42.167527    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5tnsl_f4c07b8d-4964-4899-a3ff-5f9b0fa13728/serve-healthcheck-canary/0.log"
Jun 15 05:51:42.246141 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:42.246103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-7b994bc59-tl67r" event={"ID":"0d519d85-bcda-4a59-a2e2-92e6e0e16dae","Type":"ContainerStarted","Data":"c43e1feb75de990a29a2bb7ba3526c59cbefecfb3f617a72b803bf44f4c03a0e"}
Jun 15 05:51:42.264158 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:42.264109    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-7b994bc59-tl67r" podStartSLOduration=1.684296963 podStartE2EDuration="3.264095368s" totalImagesPullingTime="1.579798405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:51:42.262723963 +0000 UTC m=+245.215590907" watchObservedRunningTime="2026-06-15 05:51:42.264095368 +0000 UTC m=+245.216962313"
Jun 15 05:51:42.767938 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:42.767909    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d4bcd4867-9wjqm_9bab946d-a7ea-473e-9e9e-00389436e017/migrator/0.log"
Jun 15 05:51:42.971354 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:42.971322    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d4bcd4867-9wjqm_9bab946d-a7ea-473e-9e9e-00389436e017/graceful-termination/0.log"
Jun 15 05:51:45.579576 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:45.579532    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:45.579576 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:45.579577    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:45.580016 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:45.579677    2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jun 15 05:51:45.580016 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:45.579678    2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:45.580016 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:45.579733    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:53.579720587 +0000 UTC m=+256.532587513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : configmap "kube-rbac-proxy" not found
Jun 15 05:51:45.580016 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:45.579750    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls podName:5861fb58-e417-4d6d-8720-81f66fbca9d4 nodeName:}" failed. No retries permitted until 2026-06-15 05:51:53.57974109 +0000 UTC m=+256.532608012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvd67" (UID: "5861fb58-e417-4d6d-8720-81f66fbca9d4") : secret "insights-runtime-extractor-tls" not found
Jun 15 05:51:47.796952 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:47.796893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:51:47.797368 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:47.797040    2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:47.797368 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:47.797120    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls podName:0bde2cab-f15d-4308-b2e7-5c02547676a3 nodeName:}" failed. No retries permitted until 2026-06-15 05:52:03.797101674 +0000 UTC m=+266.749968621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-f9d7df769-d45f6" (UID: "0bde2cab-f15d-4308-b2e7-5c02547676a3") : secret "cluster-monitoring-operator-tls" not found
Jun 15 05:51:47.898297 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:47.898256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:47.900766 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:47.900734    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"image-registry-786c49c76b-6n788\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") " pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:48.018082 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:48.018051    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g2hj5\""
Jun 15 05:51:48.025615 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:48.025587    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:48.164488 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:48.164458    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-786c49c76b-6n788"]
Jun 15 05:51:48.167729 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:51:48.167704    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1dd848_9482_467a_aefc_0c4dd0da0f5b.slice/crio-eac9b056ecb2b69b2342f09b3dd5f77180eb9a7715baca607a6ba7470949e900 WatchSource:0}: Error finding container eac9b056ecb2b69b2342f09b3dd5f77180eb9a7715baca607a6ba7470949e900: Status 404 returned error can't find the container with id eac9b056ecb2b69b2342f09b3dd5f77180eb9a7715baca607a6ba7470949e900
Jun 15 05:51:48.266246 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:48.266200    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-786c49c76b-6n788" event={"ID":"ba1dd848-9482-467a-aefc-0c4dd0da0f5b","Type":"ContainerStarted","Data":"8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c"}
Jun 15 05:51:48.266246 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:48.266249    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-786c49c76b-6n788" event={"ID":"ba1dd848-9482-467a-aefc-0c4dd0da0f5b","Type":"ContainerStarted","Data":"eac9b056ecb2b69b2342f09b3dd5f77180eb9a7715baca607a6ba7470949e900"}
Jun 15 05:51:48.266522 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:48.266338    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:51:48.293015 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:48.292963    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-786c49c76b-6n788" podStartSLOduration=16.29294602 podStartE2EDuration="16.29294602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:51:48.292161944 +0000 UTC m=+251.245028891" watchObservedRunningTime="2026-06-15 05:51:48.29294602 +0000 UTC m=+251.245812980"
Jun 15 05:51:49.410204 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:49.410167    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:51:49.410664 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:49.410562    2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jun 15 05:51:49.410664 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:51:49.410637    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs podName:7139966e-e862-401e-bf2b-f13f4f334356 nodeName:}" failed. No retries permitted until 2026-06-15 05:53:51.410619392 +0000 UTC m=+374.363486319 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs") pod "network-metrics-daemon-cplw2" (UID: "7139966e-e862-401e-bf2b-f13f4f334356") : secret "metrics-daemon-secret" not found
Jun 15 05:51:53.645978 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:53.645937    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:53.645978 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:53.645981    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:53.646580 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:53.646562    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5861fb58-e417-4d6d-8720-81f66fbca9d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:53.648483 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:53.648462    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5861fb58-e417-4d6d-8720-81f66fbca9d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvd67\" (UID: \"5861fb58-e417-4d6d-8720-81f66fbca9d4\") " pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:53.696367 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:53.696332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dvd67"
Jun 15 05:51:53.829758 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:53.829726    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dvd67"]
Jun 15 05:51:53.833715 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:51:53.833686    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5861fb58_e417_4d6d_8720_81f66fbca9d4.slice/crio-af4acc2afab5f0bf65cfe3531c157a793c94ec8de7383a6f1dbae726d665fbb7 WatchSource:0}: Error finding container af4acc2afab5f0bf65cfe3531c157a793c94ec8de7383a6f1dbae726d665fbb7: Status 404 returned error can't find the container with id af4acc2afab5f0bf65cfe3531c157a793c94ec8de7383a6f1dbae726d665fbb7
Jun 15 05:51:54.288950 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:54.288854    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvd67" event={"ID":"5861fb58-e417-4d6d-8720-81f66fbca9d4","Type":"ContainerStarted","Data":"af4acc2afab5f0bf65cfe3531c157a793c94ec8de7383a6f1dbae726d665fbb7"}
Jun 15 05:51:57.299539 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:57.299505    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvd67" event={"ID":"5861fb58-e417-4d6d-8720-81f66fbca9d4","Type":"ContainerStarted","Data":"bf5771393d58686273966ecd8e5772d29f31eed9d3cfb3615d6a2b95ebfb7f81"}
Jun 15 05:51:58.305243 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:51:58.305204    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvd67" event={"ID":"5861fb58-e417-4d6d-8720-81f66fbca9d4","Type":"ContainerStarted","Data":"2a9d480bc72ed948a24ae4cbaac495a4f69bbbaa0558fa7dba37c337b45a0b87"}
Jun 15 05:52:00.313698 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:00.313663    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvd67" event={"ID":"5861fb58-e417-4d6d-8720-81f66fbca9d4","Type":"ContainerStarted","Data":"986eaeb14661e36379c07c0b163fb78aef3e466185f80a9d242232761f097d30"}
Jun 15 05:52:00.336200 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:00.336158    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dvd67" podStartSLOduration=17.816664908 podStartE2EDuration="23.336144116s" totalImagesPullingTime="5.519479208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:51:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:00.335168288 +0000 UTC m=+263.288035235" watchObservedRunningTime="2026-06-15 05:52:00.336144116 +0000 UTC m=+263.289011060"
Jun 15 05:52:01.029909 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.029872    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-786c49c76b-6n788"]
Jun 15 05:52:01.070429 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.070372    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-65f46bd878-q4dd2"]
Jun 15 05:52:01.073509 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.073488    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.091517 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.091492    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f46bd878-q4dd2"]
Jun 15 05:52:01.107770 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.107740    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a330b581-0200-4f0e-8eb7-683a8e06cb04-installation-pull-secrets\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.107770 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.107781    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-registry-tls\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.107947 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.107808    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-bound-sa-token\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.107947 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.107894    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a330b581-0200-4f0e-8eb7-683a8e06cb04-image-registry-private-configuration\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.107947 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.107938    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a330b581-0200-4f0e-8eb7-683a8e06cb04-registry-certificates\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.108084 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.107960    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htt4t\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-kube-api-access-htt4t\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.108084 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.107979    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a330b581-0200-4f0e-8eb7-683a8e06cb04-ca-trust-extracted\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.108084 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.108026    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a330b581-0200-4f0e-8eb7-683a8e06cb04-trusted-ca\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208336 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-bound-sa-token\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208524 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208365    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a330b581-0200-4f0e-8eb7-683a8e06cb04-image-registry-private-configuration\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208524 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208408    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a330b581-0200-4f0e-8eb7-683a8e06cb04-registry-certificates\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208524 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htt4t\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-kube-api-access-htt4t\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208524 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208446    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a330b581-0200-4f0e-8eb7-683a8e06cb04-ca-trust-extracted\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208524 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a330b581-0200-4f0e-8eb7-683a8e06cb04-trusted-ca\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208524 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a330b581-0200-4f0e-8eb7-683a8e06cb04-installation-pull-secrets\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208524 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208526    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-registry-tls\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.208999 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.208971    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a330b581-0200-4f0e-8eb7-683a8e06cb04-ca-trust-extracted\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.209357 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.209276    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a330b581-0200-4f0e-8eb7-683a8e06cb04-registry-certificates\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.209912 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.209885    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a330b581-0200-4f0e-8eb7-683a8e06cb04-trusted-ca\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.211145 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.211123    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a330b581-0200-4f0e-8eb7-683a8e06cb04-installation-pull-secrets\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.211252 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.211142    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-registry-tls\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.211252 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.211126    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a330b581-0200-4f0e-8eb7-683a8e06cb04-image-registry-private-configuration\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.218322 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.218293    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-bound-sa-token\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.218856 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.218836    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htt4t\" (UniqueName: \"kubernetes.io/projected/a330b581-0200-4f0e-8eb7-683a8e06cb04-kube-api-access-htt4t\") pod \"image-registry-65f46bd878-q4dd2\" (UID: \"a330b581-0200-4f0e-8eb7-683a8e06cb04\") " pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.382409 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.382293    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:01.566665 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:52:01.566633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda330b581_0200_4f0e_8eb7_683a8e06cb04.slice/crio-c3d94f72688cf0d1fc521048ec7393f8805ec1cb58644d0d59eaea9193a8f574 WatchSource:0}: Error finding container c3d94f72688cf0d1fc521048ec7393f8805ec1cb58644d0d59eaea9193a8f574: Status 404 returned error can't find the container with id c3d94f72688cf0d1fc521048ec7393f8805ec1cb58644d0d59eaea9193a8f574
Jun 15 05:52:01.571687 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:01.571662    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-65f46bd878-q4dd2"]
Jun 15 05:52:02.322577 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:02.322543    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2" event={"ID":"a330b581-0200-4f0e-8eb7-683a8e06cb04","Type":"ContainerStarted","Data":"87fb3c6f0dd56c6efdd80c396489edc96e5e94cfd100aca09d951254ab47007a"}
Jun 15 05:52:02.322782 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:02.322594    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2" event={"ID":"a330b581-0200-4f0e-8eb7-683a8e06cb04","Type":"ContainerStarted","Data":"c3d94f72688cf0d1fc521048ec7393f8805ec1cb58644d0d59eaea9193a8f574"}
Jun 15 05:52:02.322782 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:02.322629    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:02.347358 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:02.347312    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2" podStartSLOduration=1.347296775 podStartE2EDuration="1.347296775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:02.345808039 +0000 UTC m=+265.298674995" watchObservedRunningTime="2026-06-15 05:52:02.347296775 +0000 UTC m=+265.300163720"
Jun 15 05:52:03.832189 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:03.832139    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:52:03.834698 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:03.834676    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0bde2cab-f15d-4308-b2e7-5c02547676a3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-f9d7df769-d45f6\" (UID: \"0bde2cab-f15d-4308-b2e7-5c02547676a3\") " pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:52:04.124858 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:04.124775    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-27ww5\""
Jun 15 05:52:04.132196 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:04.132176    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"
Jun 15 05:52:04.264511 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:04.264479    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6"]
Jun 15 05:52:04.267507 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:52:04.267480    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bde2cab_f15d_4308_b2e7_5c02547676a3.slice/crio-dfe1d270fbf0103926f9c2e8ec5239e41ea267fe3ff241f53298bc70db82c5c3 WatchSource:0}: Error finding container dfe1d270fbf0103926f9c2e8ec5239e41ea267fe3ff241f53298bc70db82c5c3: Status 404 returned error can't find the container with id dfe1d270fbf0103926f9c2e8ec5239e41ea267fe3ff241f53298bc70db82c5c3
Jun 15 05:52:04.331133 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:04.331097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6" event={"ID":"0bde2cab-f15d-4308-b2e7-5c02547676a3","Type":"ContainerStarted","Data":"dfe1d270fbf0103926f9c2e8ec5239e41ea267fe3ff241f53298bc70db82c5c3"}
Jun 15 05:52:06.340036 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.340001    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6" event={"ID":"0bde2cab-f15d-4308-b2e7-5c02547676a3","Type":"ContainerStarted","Data":"e93b43d71106a89a60582301802bb90ed15250d1df4bb1f2ddd7b91a4c93312c"}
Jun 15 05:52:06.355903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.355873    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"]
Jun 15 05:52:06.358867 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.358839    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"
Jun 15 05:52:06.359320 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.359283    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-f9d7df769-d45f6" podStartSLOduration=33.765503896 podStartE2EDuration="35.359272027s" totalImagesPullingTime="1.593768131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:06.358165778 +0000 UTC m=+269.311032737" watchObservedRunningTime="2026-06-15 05:52:06.359272027 +0000 UTC m=+269.312138972"
Jun 15 05:52:06.361268 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.361246    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jun 15 05:52:06.361776 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.361760    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-khr8m\""
Jun 15 05:52:06.369649 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.369630    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"]
Jun 15 05:52:06.450998 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.450950    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a1aa4784-b609-4da2-b96e-479cd17ab850-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-xrrb7\" (UID: \"a1aa4784-b609-4da2-b96e-479cd17ab850\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"
Jun 15 05:52:06.552310 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:06.552272    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a1aa4784-b609-4da2-b96e-479cd17ab850-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-xrrb7\" (UID: \"a1aa4784-b609-4da2-b96e-479cd17ab850\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"
Jun 15 05:52:06.552510 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:52:06.552403    2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jun 15 05:52:06.552510 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:52:06.552458    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1aa4784-b609-4da2-b96e-479cd17ab850-tls-certificates podName:a1aa4784-b609-4da2-b96e-479cd17ab850 nodeName:}" failed. No retries permitted until 2026-06-15 05:52:07.052443461 +0000 UTC m=+270.005310383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a1aa4784-b609-4da2-b96e-479cd17ab850-tls-certificates") pod "prometheus-operator-admission-webhook-5cf667749c-xrrb7" (UID: "a1aa4784-b609-4da2-b96e-479cd17ab850") : secret "prometheus-operator-admission-webhook-tls" not found
Jun 15 05:52:07.056002 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:07.055962    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a1aa4784-b609-4da2-b96e-479cd17ab850-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-xrrb7\" (UID: \"a1aa4784-b609-4da2-b96e-479cd17ab850\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"
Jun 15 05:52:07.058545 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:07.058523    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a1aa4784-b609-4da2-b96e-479cd17ab850-tls-certificates\") pod \"prometheus-operator-admission-webhook-5cf667749c-xrrb7\" (UID: \"a1aa4784-b609-4da2-b96e-479cd17ab850\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"
Jun 15 05:52:07.267580 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:07.267542    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"
Jun 15 05:52:07.409297 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:07.409265    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"]
Jun 15 05:52:07.412515 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:52:07.412490    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1aa4784_b609_4da2_b96e_479cd17ab850.slice/crio-165a29848fe1e7a1285721e743a3196936742bb887286888f15b0254ea72a9a1 WatchSource:0}: Error finding container 165a29848fe1e7a1285721e743a3196936742bb887286888f15b0254ea72a9a1: Status 404 returned error can't find the container with id 165a29848fe1e7a1285721e743a3196936742bb887286888f15b0254ea72a9a1
Jun 15 05:52:08.353134 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:08.352961    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7" event={"ID":"a1aa4784-b609-4da2-b96e-479cd17ab850","Type":"ContainerStarted","Data":"165a29848fe1e7a1285721e743a3196936742bb887286888f15b0254ea72a9a1"}
Jun 15 05:52:09.357733 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:09.357704    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7" event={"ID":"a1aa4784-b609-4da2-b96e-479cd17ab850","Type":"ContainerStarted","Data":"e16dc0c7e1469b1bd55a529bfc67469fb90e1c8ca48e2f0931689fe7540337d7"}
Jun 15 05:52:09.358079 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:09.357887    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"
Jun 15 05:52:09.362525 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:09.362503    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7"
Jun 15 05:52:09.385657 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:09.385623    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5cf667749c-xrrb7" podStartSLOduration=2.326110149 podStartE2EDuration="3.385608848s" totalImagesPullingTime="1.059498699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:09.384956997 +0000 UTC m=+272.337823941" watchObservedRunningTime="2026-06-15 05:52:09.385608848 +0000 UTC m=+272.338475784"
Jun 15 05:52:11.035323 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:11.035286    2562 patch_prober.go:28] interesting pod/image-registry-786c49c76b-6n788 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 15 05:52:11.035723 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:11.035343    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-786c49c76b-6n788" podUID="ba1dd848-9482-467a-aefc-0c4dd0da0f5b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:52:14.873757 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.873721    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w9mgx"]
Jun 15 05:52:14.878456 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.878429    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.881038 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.881012    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 15 05:52:14.881161 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.881072    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 15 05:52:14.881161 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.881134    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zztp9\""
Jun 15 05:52:14.881363 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.881347    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 15 05:52:14.882280 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.882254    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 15 05:52:14.921153 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921130    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-sys\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.921288 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921162    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-accelerators-collector-config\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.921288 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921188    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcvnw\" (UniqueName: \"kubernetes.io/projected/98905fcb-4803-43a5-a34a-00ae8a407f70-kube-api-access-tcvnw\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.921288 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-wtmp\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.921288 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921255    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-root\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.921480 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921316    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-textfile\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.921480 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921335    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-tls\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.921480 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921401    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:14.921480 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:14.921422    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98905fcb-4803-43a5-a34a-00ae8a407f70-metrics-client-ca\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022462 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022431    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-textfile\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022643 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022473    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-tls\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022643 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022502    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022643 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022522    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98905fcb-4803-43a5-a34a-00ae8a407f70-metrics-client-ca\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022643 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022564    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-sys\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022643 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022591    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-accelerators-collector-config\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022643 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022629    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcvnw\" (UniqueName: \"kubernetes.io/projected/98905fcb-4803-43a5-a34a-00ae8a407f70-kube-api-access-tcvnw\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022943 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022656    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-wtmp\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022943 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022670    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-sys\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022943 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022683    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-root\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022943 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022717    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-root\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022943 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022851    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-textfile\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.022943 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.022879    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-wtmp\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.023319 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.023287    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98905fcb-4803-43a5-a34a-00ae8a407f70-metrics-client-ca\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.023369 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.023349    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-accelerators-collector-config\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.025142 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.025121    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-tls\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.025231 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.025204    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98905fcb-4803-43a5-a34a-00ae8a407f70-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.054650 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.054617    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcvnw\" (UniqueName: \"kubernetes.io/projected/98905fcb-4803-43a5-a34a-00ae8a407f70-kube-api-access-tcvnw\") pod \"node-exporter-w9mgx\" (UID: \"98905fcb-4803-43a5-a34a-00ae8a407f70\") " pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.193703 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.193620    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w9mgx"
Jun 15 05:52:15.205167 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:52:15.205139    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98905fcb_4803_43a5_a34a_00ae8a407f70.slice/crio-1181b2f613a5228740ef5a0261ab74ce8e605bcfc2791a8a30ce3ab64efd0901 WatchSource:0}: Error finding container 1181b2f613a5228740ef5a0261ab74ce8e605bcfc2791a8a30ce3ab64efd0901: Status 404 returned error can't find the container with id 1181b2f613a5228740ef5a0261ab74ce8e605bcfc2791a8a30ce3ab64efd0901
Jun 15 05:52:15.378141 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:15.378111    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w9mgx" event={"ID":"98905fcb-4803-43a5-a34a-00ae8a407f70","Type":"ContainerStarted","Data":"1181b2f613a5228740ef5a0261ab74ce8e605bcfc2791a8a30ce3ab64efd0901"}
Jun 15 05:52:16.384257 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:16.384226    2562 generic.go:356] "Generic (PLEG): container finished" podID="98905fcb-4803-43a5-a34a-00ae8a407f70" containerID="b61ad72db4673bc702fcfcb43ec8164873402bda0cf278072bfed059f7cf479c" exitCode=0
Jun 15 05:52:16.384661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:16.384264    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w9mgx" event={"ID":"98905fcb-4803-43a5-a34a-00ae8a407f70","Type":"ContainerDied","Data":"b61ad72db4673bc702fcfcb43ec8164873402bda0cf278072bfed059f7cf479c"}
Jun 15 05:52:17.186242 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.186208    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-c4c58f76c-q7wgd"]
Jun 15 05:52:17.189118 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.189102    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.191966 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.191945    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 15 05:52:17.193208 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.193028    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jun 15 05:52:17.193208 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.193044    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jun 15 05:52:17.193208 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.193089    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jun 15 05:52:17.193208 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.193097    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4njqx\""
Jun 15 05:52:17.193208 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.193097    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jun 15 05:52:17.193208 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.193033    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 15 05:52:17.193558 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.193402    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jun 15 05:52:17.198914 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.198898    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jun 15 05:52:17.205088 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.205067    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4c58f76c-q7wgd"]
Jun 15 05:52:17.242269 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.242240    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-oauth-serving-cert\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.242269 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.242270    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-config\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.242430 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.242293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-oauth-config\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.242430 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.242359    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-service-ca\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.242430 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.242407    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4299q\" (UniqueName: \"kubernetes.io/projected/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-kube-api-access-4299q\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.242528 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.242451    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-trusted-ca-bundle\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.242528 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.242484    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-serving-cert\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.343416 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.343358    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-oauth-serving-cert\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.343416 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.343420    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-config\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.343596 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.343441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-oauth-config\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.343596 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.343479    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-service-ca\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.343596 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.343494    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4299q\" (UniqueName: \"kubernetes.io/projected/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-kube-api-access-4299q\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.343596 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.343550    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-trusted-ca-bundle\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.343596 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.343587    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-serving-cert\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.344298 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.344209    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-oauth-serving-cert\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.344298 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.344231    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-service-ca\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.344587 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.344375    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-config\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.344587 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.344461    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-trusted-ca-bundle\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.346093 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.346076    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-oauth-config\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.346323 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.346303    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-serving-cert\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.353369 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.353348    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4299q\" (UniqueName: \"kubernetes.io/projected/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-kube-api-access-4299q\") pod \"console-c4c58f76c-q7wgd\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") " pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.389863 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.389832    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w9mgx" event={"ID":"98905fcb-4803-43a5-a34a-00ae8a407f70","Type":"ContainerStarted","Data":"51270d7f0b3b17f23e85661c1c6eef8c006b11c0c9fed6c68dcc766f760c416a"}
Jun 15 05:52:17.390192 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.389879    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w9mgx" event={"ID":"98905fcb-4803-43a5-a34a-00ae8a407f70","Type":"ContainerStarted","Data":"6caefc2d59b087934e48ca34724a97f7f044d4a3ddd924a855c81771a76b8c43"}
Jun 15 05:52:17.414548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.414506    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w9mgx" podStartSLOduration=2.390282167 podStartE2EDuration="3.414492883s" totalImagesPullingTime="953.015088ms" totalInitContainerRuntime="71.195628ms" isStatefulPod=true podCreationTimestamp="2026-06-15 05:52:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:17.41331193 +0000 UTC m=+280.366178875" watchObservedRunningTime="2026-06-15 05:52:17.414492883 +0000 UTC m=+280.367359877"
Jun 15 05:52:17.499466 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.499433    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:17.634531 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:17.634468    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c4c58f76c-q7wgd"]
Jun 15 05:52:17.637104 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:52:17.637071    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda5d0f1d_c308_4219_9e1b_f4d6d268bd26.slice/crio-00c930469152be4d0a71f51d10030b164f8758dac555a318bb618d235e2a3494 WatchSource:0}: Error finding container 00c930469152be4d0a71f51d10030b164f8758dac555a318bb618d235e2a3494: Status 404 returned error can't find the container with id 00c930469152be4d0a71f51d10030b164f8758dac555a318bb618d235e2a3494
Jun 15 05:52:18.394204 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:18.394167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4c58f76c-q7wgd" event={"ID":"da5d0f1d-c308-4219-9e1b-f4d6d268bd26","Type":"ContainerStarted","Data":"00c930469152be4d0a71f51d10030b164f8758dac555a318bb618d235e2a3494"}
Jun 15 05:52:18.991723 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:52:18.991686    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zwqwd" podUID="30915d28-dba8-4a63-a4f6-367fe7a59f4a"
Jun 15 05:52:19.397795 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:19.397709    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:52:21.034221 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:21.034186    2562 patch_prober.go:28] interesting pod/image-registry-786c49c76b-6n788 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 15 05:52:21.034604 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:21.034250    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-786c49c76b-6n788" podUID="ba1dd848-9482-467a-aefc-0c4dd0da0f5b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:52:21.386891 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:21.386810    2562 patch_prober.go:28] interesting pod/image-registry-65f46bd878-q4dd2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 15 05:52:21.386891 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:21.386860    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2" podUID="a330b581-0200-4f0e-8eb7-683a8e06cb04" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:52:21.406440 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:21.406406    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4c58f76c-q7wgd" event={"ID":"da5d0f1d-c308-4219-9e1b-f4d6d268bd26","Type":"ContainerStarted","Data":"d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef"}
Jun 15 05:52:21.429484 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:21.429436    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-c4c58f76c-q7wgd" podStartSLOduration=1.42195529 podStartE2EDuration="4.4294223s" totalImagesPullingTime="3.00746701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:21.428436093 +0000 UTC m=+284.381303064" watchObservedRunningTime="2026-06-15 05:52:21.4294223 +0000 UTC m=+284.382289245"
Jun 15 05:52:22.284023 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:22.283983    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:52:22.286364 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:22.286345    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/30915d28-dba8-4a63-a4f6-367fe7a59f4a-metrics-tls\") pod \"dns-default-zwqwd\" (UID: \"30915d28-dba8-4a63-a4f6-367fe7a59f4a\") " pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:52:22.405161 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:22.405131    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mfr5\""
Jun 15 05:52:22.409280 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:22.409240    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:52:22.542009 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:22.541925    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zwqwd"]
Jun 15 05:52:22.544755 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:52:22.544729    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30915d28_dba8_4a63_a4f6_367fe7a59f4a.slice/crio-2ad0a68bff84c77a93aec7bc59049f8164d5ccf3f8e6aa79ed4788f83f0ef556 WatchSource:0}: Error finding container 2ad0a68bff84c77a93aec7bc59049f8164d5ccf3f8e6aa79ed4788f83f0ef556: Status 404 returned error can't find the container with id 2ad0a68bff84c77a93aec7bc59049f8164d5ccf3f8e6aa79ed4788f83f0ef556
Jun 15 05:52:23.333123 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:23.333080    2562 patch_prober.go:28] interesting pod/image-registry-65f46bd878-q4dd2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 15 05:52:23.333546 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:23.333134    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2" podUID="a330b581-0200-4f0e-8eb7-683a8e06cb04" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:52:23.415591 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:23.415554    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwqwd" event={"ID":"30915d28-dba8-4a63-a4f6-367fe7a59f4a","Type":"ContainerStarted","Data":"2ad0a68bff84c77a93aec7bc59049f8164d5ccf3f8e6aa79ed4788f83f0ef556"}
Jun 15 05:52:24.217694 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.217113    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57874654c-xdks7"]
Jun 15 05:52:24.220809 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.220788    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.232202 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.232138    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57874654c-xdks7"]
Jun 15 05:52:24.301623 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.301598    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w92rz\" (UniqueName: \"kubernetes.io/projected/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-kube-api-access-w92rz\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.301775 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.301631    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-oauth-config\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.301775 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.301648    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-trusted-ca-bundle\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.301775 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.301734    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-config\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.301898 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.301782    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-service-ca\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.301898 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.301805    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-oauth-serving-cert\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.301898 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.301871    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-serving-cert\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.402267 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.402181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-serving-cert\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.402267 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.402223    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w92rz\" (UniqueName: \"kubernetes.io/projected/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-kube-api-access-w92rz\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.402267 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.402249    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-oauth-config\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.402822 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.402274    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-trusted-ca-bundle\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.402822 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.402323    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-config\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.402822 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.402370    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-service-ca\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.402822 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.402427    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-oauth-serving-cert\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.403158 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.403134    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-oauth-serving-cert\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.403271 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.403248    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-trusted-ca-bundle\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.403315 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.403248    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-service-ca\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.403428 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.403408    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-config\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.404969 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.404943    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-oauth-config\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.405049 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.405005    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-serving-cert\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.419028 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.419005    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w92rz\" (UniqueName: \"kubernetes.io/projected/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-kube-api-access-w92rz\") pod \"console-57874654c-xdks7\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") " pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.420429 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.420402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwqwd" event={"ID":"30915d28-dba8-4a63-a4f6-367fe7a59f4a","Type":"ContainerStarted","Data":"d624c131b73d852f4b52fe3cec34eca0244885604a2c9bdd1d660371858863f2"}
Jun 15 05:52:24.420511 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.420464    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zwqwd" event={"ID":"30915d28-dba8-4a63-a4f6-367fe7a59f4a","Type":"ContainerStarted","Data":"a95f3866f1307b182c40a165dd2e52218712c3423290d8d95198d24d3e196a1c"}
Jun 15 05:52:24.420564 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.420549    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:52:24.444878 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.444841    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zwqwd" podStartSLOduration=250.892551416 podStartE2EDuration="4m12.444826003s" totalImagesPullingTime="1.552274587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:24.44327228 +0000 UTC m=+287.396139236" watchObservedRunningTime="2026-06-15 05:52:24.444826003 +0000 UTC m=+287.397692939"
Jun 15 05:52:24.534478 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.534440    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:24.668083 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:24.668011    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57874654c-xdks7"]
Jun 15 05:52:24.670913 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:52:24.670889    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c3c49f_b096_4cf0_a5cb_0149346a9b9c.slice/crio-a6b38b57bf01af5b3f7cf223793d4ce842ed9330c0687df1834c596c8c39ff63 WatchSource:0}: Error finding container a6b38b57bf01af5b3f7cf223793d4ce842ed9330c0687df1834c596c8c39ff63: Status 404 returned error can't find the container with id a6b38b57bf01af5b3f7cf223793d4ce842ed9330c0687df1834c596c8c39ff63
Jun 15 05:52:25.425495 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:25.425458    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57874654c-xdks7" event={"ID":"99c3c49f-b096-4cf0-a5cb-0149346a9b9c","Type":"ContainerStarted","Data":"385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4"}
Jun 15 05:52:25.425876 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:25.425515    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57874654c-xdks7" event={"ID":"99c3c49f-b096-4cf0-a5cb-0149346a9b9c","Type":"ContainerStarted","Data":"a6b38b57bf01af5b3f7cf223793d4ce842ed9330c0687df1834c596c8c39ff63"}
Jun 15 05:52:25.447566 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:25.447520    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57874654c-xdks7" podStartSLOduration=1.447503964 podStartE2EDuration="1.447503964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:52:25.446050738 +0000 UTC m=+288.398917676" watchObservedRunningTime="2026-06-15 05:52:25.447503964 +0000 UTC m=+288.400370905"
Jun 15 05:52:26.049520 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.049476    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-786c49c76b-6n788" podUID="ba1dd848-9482-467a-aefc-0c4dd0da0f5b" containerName="registry" containerID="cri-o://8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c" gracePeriod=30
Jun 15 05:52:26.295714 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.295692    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:52:26.420738 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.420658    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-bound-sa-token\") pod \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") "
Jun 15 05:52:26.420738 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.420713    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-certificates\") pod \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") "
Jun 15 05:52:26.420966 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.420739    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-installation-pull-secrets\") pod \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") "
Jun 15 05:52:26.420966 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.420770    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-trusted-ca\") pod \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") "
Jun 15 05:52:26.420966 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.420795    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdnr5\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-kube-api-access-tdnr5\") pod \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") "
Jun 15 05:52:26.420966 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.420839    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-ca-trust-extracted\") pod \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") "
Jun 15 05:52:26.420966 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.420882    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") pod \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") "
Jun 15 05:52:26.420966 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.420912    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-image-registry-private-configuration\") pod \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\" (UID: \"ba1dd848-9482-467a-aefc-0c4dd0da0f5b\") "
Jun 15 05:52:26.421314 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.421208    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ba1dd848-9482-467a-aefc-0c4dd0da0f5b" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:52:26.421314 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.421249    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ba1dd848-9482-467a-aefc-0c4dd0da0f5b" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:52:26.423226 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.423195    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ba1dd848-9482-467a-aefc-0c4dd0da0f5b" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:52:26.423422 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.423374    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ba1dd848-9482-467a-aefc-0c4dd0da0f5b" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:52:26.423574 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.423548    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ba1dd848-9482-467a-aefc-0c4dd0da0f5b" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:52:26.423574 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.423556    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-kube-api-access-tdnr5" (OuterVolumeSpecName: "kube-api-access-tdnr5") pod "ba1dd848-9482-467a-aefc-0c4dd0da0f5b" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b"). InnerVolumeSpecName "kube-api-access-tdnr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:52:26.423696 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.423637    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ba1dd848-9482-467a-aefc-0c4dd0da0f5b" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:52:26.430046 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.430018    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ba1dd848-9482-467a-aefc-0c4dd0da0f5b" (UID: "ba1dd848-9482-467a-aefc-0c4dd0da0f5b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:52:26.430995 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.430970    2562 generic.go:356] "Generic (PLEG): container finished" podID="ba1dd848-9482-467a-aefc-0c4dd0da0f5b" containerID="8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c" exitCode=0
Jun 15 05:52:26.431075 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.431015    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-786c49c76b-6n788"
Jun 15 05:52:26.431075 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.431045    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-786c49c76b-6n788" event={"ID":"ba1dd848-9482-467a-aefc-0c4dd0da0f5b","Type":"ContainerDied","Data":"8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c"}
Jun 15 05:52:26.431160 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.431092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-786c49c76b-6n788" event={"ID":"ba1dd848-9482-467a-aefc-0c4dd0da0f5b","Type":"ContainerDied","Data":"eac9b056ecb2b69b2342f09b3dd5f77180eb9a7715baca607a6ba7470949e900"}
Jun 15 05:52:26.431160 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.431111    2562 scope.go:117] "RemoveContainer" containerID="8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c"
Jun 15 05:52:26.441622 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.441596    2562 scope.go:117] "RemoveContainer" containerID="8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c"
Jun 15 05:52:26.441960 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:52:26.441929    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c\": container with ID starting with 8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c not found: ID does not exist" containerID="8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c"
Jun 15 05:52:26.441960 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.441971    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c"} err="failed to get container status \"8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c\": rpc error: code = NotFound desc = could not find container \"8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c\": container with ID starting with 8431c88df17a6bcc21c437bf3b1ec6512d7f6fd28679d6b3462a5561f1ab056c not found: ID does not exist"
Jun 15 05:52:26.458131 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.458106    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-786c49c76b-6n788"]
Jun 15 05:52:26.462369 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.462350    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-786c49c76b-6n788"]
Jun 15 05:52:26.521944 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.521903    2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-ca-trust-extracted\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:52:26.521944 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.521935    2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:52:26.521944 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.521952    2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-image-registry-private-configuration\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:52:26.522174 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.521965    2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-bound-sa-token\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:52:26.522174 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.521977    2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-registry-certificates\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:52:26.522174 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.521990    2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-installation-pull-secrets\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:52:26.522174 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.522003    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-trusted-ca\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:52:26.522174 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:26.522015    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnr5\" (UniqueName: \"kubernetes.io/projected/ba1dd848-9482-467a-aefc-0c4dd0da0f5b-kube-api-access-tdnr5\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:52:27.488860 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:27.488823    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba1dd848-9482-467a-aefc-0c4dd0da0f5b" path="/var/lib/kubelet/pods/ba1dd848-9482-467a-aefc-0c4dd0da0f5b/volumes"
Jun 15 05:52:27.500143 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:27.500115    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:27.500266 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:27.500178    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:27.504792 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:27.504767    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:28.442615 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:28.442588    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:52:31.386603 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:31.386570    2562 patch_prober.go:28] interesting pod/image-registry-65f46bd878-q4dd2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jun 15 05:52:31.387050 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:31.386620    2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2" podUID="a330b581-0200-4f0e-8eb7-683a8e06cb04" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:52:32.484520 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:52:32.484488    2562 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cplw2" podUID="7139966e-e862-401e-bf2b-f13f4f334356"
Jun 15 05:52:33.331156 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:33.331124    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-65f46bd878-q4dd2"
Jun 15 05:52:34.428098 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:34.428065    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zwqwd"
Jun 15 05:52:34.535220 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:34.535188    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:34.535392 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:34.535245    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:34.539693 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:34.539676    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:35.467042 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:35.467013    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:52:35.551062 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:35.551029    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c4c58f76c-q7wgd"]
Jun 15 05:52:37.436032 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:37.436005    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:52:37.436567 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:37.436547    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:52:37.442798 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:37.442782    2562 kubelet.go:1648] "Image garbage collection succeeded"
Jun 15 05:52:47.485853 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:52:47.485813    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:53:00.570617 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.570578    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-c4c58f76c-q7wgd" podUID="da5d0f1d-c308-4219-9e1b-f4d6d268bd26" containerName="console" containerID="cri-o://d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef" gracePeriod=15
Jun 15 05:53:00.804679 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.804657    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:53:00.894790 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.894707    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-oauth-config\") pod \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") "
Jun 15 05:53:00.894790 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.894748    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-trusted-ca-bundle\") pod \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") "
Jun 15 05:53:00.895005 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.894881    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-config\") pod \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") "
Jun 15 05:53:00.895005 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.894918    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-service-ca\") pod \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") "
Jun 15 05:53:00.895005 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.894947    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-oauth-serving-cert\") pod \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") "
Jun 15 05:53:00.895005 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.894973    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-serving-cert\") pod \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") "
Jun 15 05:53:00.895190 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.895038    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4299q\" (UniqueName: \"kubernetes.io/projected/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-kube-api-access-4299q\") pod \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\" (UID: \"da5d0f1d-c308-4219-9e1b-f4d6d268bd26\") "
Jun 15 05:53:00.895190 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.895149    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "da5d0f1d-c308-4219-9e1b-f4d6d268bd26" (UID: "da5d0f1d-c308-4219-9e1b-f4d6d268bd26"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:53:00.895284 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.895269    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-service-ca" (OuterVolumeSpecName: "service-ca") pod "da5d0f1d-c308-4219-9e1b-f4d6d268bd26" (UID: "da5d0f1d-c308-4219-9e1b-f4d6d268bd26"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:53:00.895340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.895280    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-trusted-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:53:00.895340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.895282    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-config" (OuterVolumeSpecName: "console-config") pod "da5d0f1d-c308-4219-9e1b-f4d6d268bd26" (UID: "da5d0f1d-c308-4219-9e1b-f4d6d268bd26"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:53:00.895340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.895309    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "da5d0f1d-c308-4219-9e1b-f4d6d268bd26" (UID: "da5d0f1d-c308-4219-9e1b-f4d6d268bd26"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:53:00.897248 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.897227    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-kube-api-access-4299q" (OuterVolumeSpecName: "kube-api-access-4299q") pod "da5d0f1d-c308-4219-9e1b-f4d6d268bd26" (UID: "da5d0f1d-c308-4219-9e1b-f4d6d268bd26"). InnerVolumeSpecName "kube-api-access-4299q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:53:00.897330 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.897252    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "da5d0f1d-c308-4219-9e1b-f4d6d268bd26" (UID: "da5d0f1d-c308-4219-9e1b-f4d6d268bd26"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:53:00.897330 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.897313    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "da5d0f1d-c308-4219-9e1b-f4d6d268bd26" (UID: "da5d0f1d-c308-4219-9e1b-f4d6d268bd26"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:53:00.996635 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.996591    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:53:00.996635 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.996621    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-service-ca\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:53:00.996635 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.996636    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-oauth-serving-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:53:00.996635 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.996646    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-serving-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:53:00.996891 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.996655    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4299q\" (UniqueName: \"kubernetes.io/projected/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-kube-api-access-4299q\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:53:00.996891 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:00.996665    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/da5d0f1d-c308-4219-9e1b-f4d6d268bd26-console-oauth-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:53:01.550125 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.550097    2562 generic.go:356] "Generic (PLEG): container finished" podID="da5d0f1d-c308-4219-9e1b-f4d6d268bd26" containerID="d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef" exitCode=0
Jun 15 05:53:01.550307 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.550143    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c4c58f76c-q7wgd"
Jun 15 05:53:01.550307 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.550167    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4c58f76c-q7wgd" event={"ID":"da5d0f1d-c308-4219-9e1b-f4d6d268bd26","Type":"ContainerDied","Data":"d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef"}
Jun 15 05:53:01.550307 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.550206    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c4c58f76c-q7wgd" event={"ID":"da5d0f1d-c308-4219-9e1b-f4d6d268bd26","Type":"ContainerDied","Data":"00c930469152be4d0a71f51d10030b164f8758dac555a318bb618d235e2a3494"}
Jun 15 05:53:01.550307 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.550225    2562 scope.go:117] "RemoveContainer" containerID="d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef"
Jun 15 05:53:01.558433 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.558398    2562 scope.go:117] "RemoveContainer" containerID="d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef"
Jun 15 05:53:01.558685 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:53:01.558667    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef\": container with ID starting with d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef not found: ID does not exist" containerID="d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef"
Jun 15 05:53:01.558738 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.558692    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef"} err="failed to get container status \"d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef\": rpc error: code = NotFound desc = could not find container \"d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef\": container with ID starting with d598a4eba0e89ad73864578ea9cc4b7d834b29a37c350b2912d1745126ed6bef not found: ID does not exist"
Jun 15 05:53:01.575311 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.575270    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c4c58f76c-q7wgd"]
Jun 15 05:53:01.579046 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:01.579008    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c4c58f76c-q7wgd"]
Jun 15 05:53:03.489126 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:03.489086    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da5d0f1d-c308-4219-9e1b-f4d6d268bd26" path="/var/lib/kubelet/pods/da5d0f1d-c308-4219-9e1b-f4d6d268bd26/volumes"
Jun 15 05:53:43.438604 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.438568    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cc6fcb4dd-pr4n9"]
Jun 15 05:53:43.439052 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.438845    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba1dd848-9482-467a-aefc-0c4dd0da0f5b" containerName="registry"
Jun 15 05:53:43.439052 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.438856    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba1dd848-9482-467a-aefc-0c4dd0da0f5b" containerName="registry"
Jun 15 05:53:43.439052 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.438872    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da5d0f1d-c308-4219-9e1b-f4d6d268bd26" containerName="console"
Jun 15 05:53:43.439052 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.438877    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="da5d0f1d-c308-4219-9e1b-f4d6d268bd26" containerName="console"
Jun 15 05:53:43.439052 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.438919    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba1dd848-9482-467a-aefc-0c4dd0da0f5b" containerName="registry"
Jun 15 05:53:43.439052 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.438928    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="da5d0f1d-c308-4219-9e1b-f4d6d268bd26" containerName="console"
Jun 15 05:53:43.442758 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.442743    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.458284 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.458260    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc6fcb4dd-pr4n9"]
Jun 15 05:53:43.529679 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.529651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-oauth-config\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.529854 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.529684    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-trusted-ca-bundle\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.529854 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.529702    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-oauth-serving-cert\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.529854 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.529743    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-serving-cert\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.529854 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.529764    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-service-ca\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.529854 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.529785    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-config\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.530040 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.529870    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjng\" (UniqueName: \"kubernetes.io/projected/88df66d3-8e04-4f5b-89bd-35a77a66059f-kube-api-access-zmjng\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.630638 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.630592    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-serving-cert\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.630638 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.630641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-service-ca\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.630871 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.630665    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-config\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.630871 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.630720    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjng\" (UniqueName: \"kubernetes.io/projected/88df66d3-8e04-4f5b-89bd-35a77a66059f-kube-api-access-zmjng\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.630871 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.630755    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-oauth-config\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.630871 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.630782    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-trusted-ca-bundle\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.630871 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.630804    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-oauth-serving-cert\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.631538 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.631510    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-oauth-serving-cert\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.631658 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.631510    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-config\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.631658 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.631610    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-trusted-ca-bundle\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.631801 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.631783    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-service-ca\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.633179 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.633162    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-serving-cert\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.633237 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.633199    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-oauth-config\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.642898 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.642874    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjng\" (UniqueName: \"kubernetes.io/projected/88df66d3-8e04-4f5b-89bd-35a77a66059f-kube-api-access-zmjng\") pod \"console-cc6fcb4dd-pr4n9\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") " pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.750997 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.750955    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:43.897254 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:43.897220    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc6fcb4dd-pr4n9"]
Jun 15 05:53:43.900489 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:53:43.900457    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88df66d3_8e04_4f5b_89bd_35a77a66059f.slice/crio-2f9b1958742c7906d8a973e072115931739d1b41d0a6111054873f1518c5cd82 WatchSource:0}: Error finding container 2f9b1958742c7906d8a973e072115931739d1b41d0a6111054873f1518c5cd82: Status 404 returned error can't find the container with id 2f9b1958742c7906d8a973e072115931739d1b41d0a6111054873f1518c5cd82
Jun 15 05:53:44.699663 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:44.699626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc6fcb4dd-pr4n9" event={"ID":"88df66d3-8e04-4f5b-89bd-35a77a66059f","Type":"ContainerStarted","Data":"069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18"}
Jun 15 05:53:44.700032 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:44.699673    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc6fcb4dd-pr4n9" event={"ID":"88df66d3-8e04-4f5b-89bd-35a77a66059f","Type":"ContainerStarted","Data":"2f9b1958742c7906d8a973e072115931739d1b41d0a6111054873f1518c5cd82"}
Jun 15 05:53:44.722285 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:44.722232    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cc6fcb4dd-pr4n9" podStartSLOduration=1.722213988 podStartE2EDuration="1.722213988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:53:44.721735834 +0000 UTC m=+367.674602771" watchObservedRunningTime="2026-06-15 05:53:44.722213988 +0000 UTC m=+367.675080936"
Jun 15 05:53:51.496026 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:51.495992    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:53:51.498468 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:51.498444    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7139966e-e862-401e-bf2b-f13f4f334356-metrics-certs\") pod \"network-metrics-daemon-cplw2\" (UID: \"7139966e-e862-401e-bf2b-f13f4f334356\") " pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:53:51.689150 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:51.689122    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8qp7w\""
Jun 15 05:53:51.696727 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:51.696704    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cplw2"
Jun 15 05:53:51.827463 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:51.827434    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cplw2"]
Jun 15 05:53:51.830636 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:53:51.830608    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7139966e_e862_401e_bf2b_f13f4f334356.slice/crio-8358ec29395dac6e85be25c7cda2c87d25f6b0ab0bb001dbdce20b4024c3bc8c WatchSource:0}: Error finding container 8358ec29395dac6e85be25c7cda2c87d25f6b0ab0bb001dbdce20b4024c3bc8c: Status 404 returned error can't find the container with id 8358ec29395dac6e85be25c7cda2c87d25f6b0ab0bb001dbdce20b4024c3bc8c
Jun 15 05:53:51.832629 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:51.832614    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:53:52.729467 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:52.729404    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cplw2" event={"ID":"7139966e-e862-401e-bf2b-f13f4f334356","Type":"ContainerStarted","Data":"8358ec29395dac6e85be25c7cda2c87d25f6b0ab0bb001dbdce20b4024c3bc8c"}
Jun 15 05:53:53.734313 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:53.734277    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cplw2" event={"ID":"7139966e-e862-401e-bf2b-f13f4f334356","Type":"ContainerStarted","Data":"a5ad54ea98c2f571d1eecc1bfd1c4e922243d971bacfeb248bba50b6360eb331"}
Jun 15 05:53:53.734792 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:53.734333    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cplw2" event={"ID":"7139966e-e862-401e-bf2b-f13f4f334356","Type":"ContainerStarted","Data":"12e9f4609f96d78e8a5e69adda8c631599a933a1edf5f9f5a3af8b723d74c54f"}
Jun 15 05:53:53.751457 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:53.751429    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:53.751661 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:53.751486    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:53.752599 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:53.752556    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cplw2" podStartSLOduration=375.685777481 podStartE2EDuration="6m16.752537442s" totalImagesPullingTime="1.066759961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:53:53.752010829 +0000 UTC m=+376.704877775" watchObservedRunningTime="2026-06-15 05:53:53.752537442 +0000 UTC m=+376.705404420"
Jun 15 05:53:53.756312 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:53.756295    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:54.742829 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:54.742799    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:53:54.800472 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:53:54.800433    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57874654c-xdks7"]
Jun 15 05:54:19.821720 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:19.821668    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-57874654c-xdks7" podUID="99c3c49f-b096-4cf0-a5cb-0149346a9b9c" containerName="console" containerID="cri-o://385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4" gracePeriod=15
Jun 15 05:54:20.073783 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.073707    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:54:20.102394 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.102359    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-oauth-config\") pod \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") "
Jun 15 05:54:20.102587 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.102436    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w92rz\" (UniqueName: \"kubernetes.io/projected/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-kube-api-access-w92rz\") pod \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") "
Jun 15 05:54:20.102587 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.102538    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-serving-cert\") pod \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") "
Jun 15 05:54:20.102703 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.102636    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-trusted-ca-bundle\") pod \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") "
Jun 15 05:54:20.102760 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.102709    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-config\") pod \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") "
Jun 15 05:54:20.102760 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.102741    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-oauth-serving-cert\") pod \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") "
Jun 15 05:54:20.102859 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.102784    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-service-ca\") pod \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\" (UID: \"99c3c49f-b096-4cf0-a5cb-0149346a9b9c\") "
Jun 15 05:54:20.103021 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.102993    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "99c3c49f-b096-4cf0-a5cb-0149346a9b9c" (UID: "99c3c49f-b096-4cf0-a5cb-0149346a9b9c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:54:20.103534 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.103504    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "99c3c49f-b096-4cf0-a5cb-0149346a9b9c" (UID: "99c3c49f-b096-4cf0-a5cb-0149346a9b9c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:54:20.103710 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.103561    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-config" (OuterVolumeSpecName: "console-config") pod "99c3c49f-b096-4cf0-a5cb-0149346a9b9c" (UID: "99c3c49f-b096-4cf0-a5cb-0149346a9b9c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:54:20.103710 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.103659    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-service-ca" (OuterVolumeSpecName: "service-ca") pod "99c3c49f-b096-4cf0-a5cb-0149346a9b9c" (UID: "99c3c49f-b096-4cf0-a5cb-0149346a9b9c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:54:20.104935 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.104904    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "99c3c49f-b096-4cf0-a5cb-0149346a9b9c" (UID: "99c3c49f-b096-4cf0-a5cb-0149346a9b9c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:54:20.105096 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.105073    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-kube-api-access-w92rz" (OuterVolumeSpecName: "kube-api-access-w92rz") pod "99c3c49f-b096-4cf0-a5cb-0149346a9b9c" (UID: "99c3c49f-b096-4cf0-a5cb-0149346a9b9c"). InnerVolumeSpecName "kube-api-access-w92rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:54:20.105096 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.105089    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "99c3c49f-b096-4cf0-a5cb-0149346a9b9c" (UID: "99c3c49f-b096-4cf0-a5cb-0149346a9b9c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:54:20.203282 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.203246    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-trusted-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:54:20.203282 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.203274    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:54:20.203282 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.203283    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-oauth-serving-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:54:20.203282 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.203291    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-service-ca\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:54:20.203568 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.203300    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-oauth-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:54:20.203568 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.203309    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w92rz\" (UniqueName: \"kubernetes.io/projected/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-kube-api-access-w92rz\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:54:20.203568 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.203320    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/99c3c49f-b096-4cf0-a5cb-0149346a9b9c-console-serving-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:54:20.832068 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.832037    2562 generic.go:356] "Generic (PLEG): container finished" podID="99c3c49f-b096-4cf0-a5cb-0149346a9b9c" containerID="385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4" exitCode=0
Jun 15 05:54:20.832598 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.832089    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57874654c-xdks7"
Jun 15 05:54:20.832598 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.832092    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57874654c-xdks7" event={"ID":"99c3c49f-b096-4cf0-a5cb-0149346a9b9c","Type":"ContainerDied","Data":"385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4"}
Jun 15 05:54:20.832598 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.832219    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57874654c-xdks7" event={"ID":"99c3c49f-b096-4cf0-a5cb-0149346a9b9c","Type":"ContainerDied","Data":"a6b38b57bf01af5b3f7cf223793d4ce842ed9330c0687df1834c596c8c39ff63"}
Jun 15 05:54:20.832598 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.832247    2562 scope.go:117] "RemoveContainer" containerID="385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4"
Jun 15 05:54:20.840862 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.840845    2562 scope.go:117] "RemoveContainer" containerID="385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4"
Jun 15 05:54:20.841086 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:54:20.841065    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4\": container with ID starting with 385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4 not found: ID does not exist" containerID="385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4"
Jun 15 05:54:20.841134 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.841095    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4"} err="failed to get container status \"385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4\": rpc error: code = NotFound desc = could not find container \"385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4\": container with ID starting with 385e4d44064de72f472e5ec7d696d1fdd1f22cb6170c7e05a4ccee4534910ef4 not found: ID does not exist"
Jun 15 05:54:20.858052 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.858027    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-57874654c-xdks7"]
Jun 15 05:54:20.863762 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:20.863738    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-57874654c-xdks7"]
Jun 15 05:54:21.488393 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:21.488341    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c3c49f-b096-4cf0-a5cb-0149346a9b9c" path="/var/lib/kubelet/pods/99c3c49f-b096-4cf0-a5cb-0149346a9b9c/volumes"
Jun 15 05:54:57.104303 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.104221    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7856cd94c7-8jd9k"]
Jun 15 05:54:57.104787 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.104563    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c3c49f-b096-4cf0-a5cb-0149346a9b9c" containerName="console"
Jun 15 05:54:57.104787 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.104579    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c3c49f-b096-4cf0-a5cb-0149346a9b9c" containerName="console"
Jun 15 05:54:57.104787 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.104631    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c3c49f-b096-4cf0-a5cb-0149346a9b9c" containerName="console"
Jun 15 05:54:57.107555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.107536    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.148399 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.148351    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7856cd94c7-8jd9k"]
Jun 15 05:54:57.166572 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.166543    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-oauth-config\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.166732 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.166589    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-serving-cert\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.166732 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.166636    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-oauth-serving-cert\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.166732 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.166655    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-config\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.166732 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.166672    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qch\" (UniqueName: \"kubernetes.io/projected/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-kube-api-access-45qch\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.166732 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.166720    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-service-ca\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.166912 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.166777    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-trusted-ca-bundle\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.267670 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.267636    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-trusted-ca-bundle\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.267863 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.267682    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-oauth-config\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.267863 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.267710    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-serving-cert\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.267863 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.267749    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-oauth-serving-cert\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.268019 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.267893    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-config\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.268019 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.267942    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qch\" (UniqueName: \"kubernetes.io/projected/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-kube-api-access-45qch\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.268019 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.267973    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-service-ca\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.268562 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.268513    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-oauth-serving-cert\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.268853 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.268824    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-config\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.268970 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.268935    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-trusted-ca-bundle\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.269221 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.269202    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-service-ca\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.270555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.270532    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-oauth-config\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.270702 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.270683    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-serving-cert\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.288610 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.288578    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qch\" (UniqueName: \"kubernetes.io/projected/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-kube-api-access-45qch\") pod \"console-7856cd94c7-8jd9k\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") " pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.416584 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.416484    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:54:57.566695 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.566667    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7856cd94c7-8jd9k"]
Jun 15 05:54:57.573009 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:54:57.572985    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a2e1cd_8b40_46e1_87fa_fb0bf4e4ca40.slice/crio-7a78bcd9d1f0024aa38ffcc8c03ce978815078daade413db6bc59e519c2d46f5 WatchSource:0}: Error finding container 7a78bcd9d1f0024aa38ffcc8c03ce978815078daade413db6bc59e519c2d46f5: Status 404 returned error can't find the container with id 7a78bcd9d1f0024aa38ffcc8c03ce978815078daade413db6bc59e519c2d46f5
Jun 15 05:54:57.968518 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.968486    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7856cd94c7-8jd9k" event={"ID":"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40","Type":"ContainerStarted","Data":"f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd"}
Jun 15 05:54:57.968700 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:57.968534    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7856cd94c7-8jd9k" event={"ID":"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40","Type":"ContainerStarted","Data":"7a78bcd9d1f0024aa38ffcc8c03ce978815078daade413db6bc59e519c2d46f5"}
Jun 15 05:54:58.017789 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:54:58.017732    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7856cd94c7-8jd9k" podStartSLOduration=1.017715016 podStartE2EDuration="1.017715016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:54:58.017364542 +0000 UTC m=+440.970231488" watchObservedRunningTime="2026-06-15 05:54:58.017715016 +0000 UTC m=+440.970581962"
Jun 15 05:55:07.417058 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:07.417022    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:55:07.417473 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:07.417124    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:55:07.421766 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:07.421747    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:55:08.008023 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:08.007987    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:55:08.109457 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:08.109427    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc6fcb4dd-pr4n9"]
Jun 15 05:55:33.129779 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.129740    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-cc6fcb4dd-pr4n9" podUID="88df66d3-8e04-4f5b-89bd-35a77a66059f" containerName="console" containerID="cri-o://069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18" gracePeriod=15
Jun 15 05:55:33.373888 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.373862    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:55:33.455458 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455354    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-serving-cert\") pod \"88df66d3-8e04-4f5b-89bd-35a77a66059f\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") "
Jun 15 05:55:33.455458 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455447    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmjng\" (UniqueName: \"kubernetes.io/projected/88df66d3-8e04-4f5b-89bd-35a77a66059f-kube-api-access-zmjng\") pod \"88df66d3-8e04-4f5b-89bd-35a77a66059f\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") "
Jun 15 05:55:33.455687 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455489    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-config\") pod \"88df66d3-8e04-4f5b-89bd-35a77a66059f\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") "
Jun 15 05:55:33.455687 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455512    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-oauth-config\") pod \"88df66d3-8e04-4f5b-89bd-35a77a66059f\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") "
Jun 15 05:55:33.455687 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455535    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-trusted-ca-bundle\") pod \"88df66d3-8e04-4f5b-89bd-35a77a66059f\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") "
Jun 15 05:55:33.455687 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455559    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-oauth-serving-cert\") pod \"88df66d3-8e04-4f5b-89bd-35a77a66059f\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") "
Jun 15 05:55:33.455687 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455579    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-service-ca\") pod \"88df66d3-8e04-4f5b-89bd-35a77a66059f\" (UID: \"88df66d3-8e04-4f5b-89bd-35a77a66059f\") "
Jun 15 05:55:33.456009 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455974    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "88df66d3-8e04-4f5b-89bd-35a77a66059f" (UID: "88df66d3-8e04-4f5b-89bd-35a77a66059f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:55:33.456069 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.455985    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-config" (OuterVolumeSpecName: "console-config") pod "88df66d3-8e04-4f5b-89bd-35a77a66059f" (UID: "88df66d3-8e04-4f5b-89bd-35a77a66059f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:55:33.456069 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.456000    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "88df66d3-8e04-4f5b-89bd-35a77a66059f" (UID: "88df66d3-8e04-4f5b-89bd-35a77a66059f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:55:33.456069 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.456042    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-service-ca" (OuterVolumeSpecName: "service-ca") pod "88df66d3-8e04-4f5b-89bd-35a77a66059f" (UID: "88df66d3-8e04-4f5b-89bd-35a77a66059f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:55:33.456359 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.456339    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:33.456429 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.456367    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-trusted-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:33.457767 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.457739    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "88df66d3-8e04-4f5b-89bd-35a77a66059f" (UID: "88df66d3-8e04-4f5b-89bd-35a77a66059f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:55:33.457767 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.457757    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "88df66d3-8e04-4f5b-89bd-35a77a66059f" (UID: "88df66d3-8e04-4f5b-89bd-35a77a66059f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:55:33.457884 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.457768    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88df66d3-8e04-4f5b-89bd-35a77a66059f-kube-api-access-zmjng" (OuterVolumeSpecName: "kube-api-access-zmjng") pod "88df66d3-8e04-4f5b-89bd-35a77a66059f" (UID: "88df66d3-8e04-4f5b-89bd-35a77a66059f"). InnerVolumeSpecName "kube-api-access-zmjng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:55:33.557067 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.557037    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-serving-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:33.557067 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.557066    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmjng\" (UniqueName: \"kubernetes.io/projected/88df66d3-8e04-4f5b-89bd-35a77a66059f-kube-api-access-zmjng\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:33.557261 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.557081    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88df66d3-8e04-4f5b-89bd-35a77a66059f-console-oauth-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:33.557261 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.557095    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-oauth-serving-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:33.557261 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:33.557108    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88df66d3-8e04-4f5b-89bd-35a77a66059f-service-ca\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:34.096927 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.096892    2562 generic.go:356] "Generic (PLEG): container finished" podID="88df66d3-8e04-4f5b-89bd-35a77a66059f" containerID="069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18" exitCode=0
Jun 15 05:55:34.097108 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.096938    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc6fcb4dd-pr4n9"
Jun 15 05:55:34.097108 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.096979    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc6fcb4dd-pr4n9" event={"ID":"88df66d3-8e04-4f5b-89bd-35a77a66059f","Type":"ContainerDied","Data":"069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18"}
Jun 15 05:55:34.097108 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.097032    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc6fcb4dd-pr4n9" event={"ID":"88df66d3-8e04-4f5b-89bd-35a77a66059f","Type":"ContainerDied","Data":"2f9b1958742c7906d8a973e072115931739d1b41d0a6111054873f1518c5cd82"}
Jun 15 05:55:34.097108 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.097052    2562 scope.go:117] "RemoveContainer" containerID="069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18"
Jun 15 05:55:34.105118 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.105101    2562 scope.go:117] "RemoveContainer" containerID="069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18"
Jun 15 05:55:34.105440 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:55:34.105410    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18\": container with ID starting with 069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18 not found: ID does not exist" containerID="069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18"
Jun 15 05:55:34.105530 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.105452    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18"} err="failed to get container status \"069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18\": rpc error: code = NotFound desc = could not find container \"069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18\": container with ID starting with 069e5e0190e7d9862547e2ebef12b424e6476b5dbfe2b4cb2cf50f6449604f18 not found: ID does not exist"
Jun 15 05:55:34.118637 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.118606    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc6fcb4dd-pr4n9"]
Jun 15 05:55:34.124080 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:34.124058    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cc6fcb4dd-pr4n9"]
Jun 15 05:55:35.489178 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:35.489148    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88df66d3-8e04-4f5b-89bd-35a77a66059f" path="/var/lib/kubelet/pods/88df66d3-8e04-4f5b-89bd-35a77a66059f/volumes"
Jun 15 05:55:39.795014 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.794973    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"]
Jun 15 05:55:39.795503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.795324    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88df66d3-8e04-4f5b-89bd-35a77a66059f" containerName="console"
Jun 15 05:55:39.795503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.795342    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="88df66d3-8e04-4f5b-89bd-35a77a66059f" containerName="console"
Jun 15 05:55:39.795503 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.795453    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="88df66d3-8e04-4f5b-89bd-35a77a66059f" containerName="console"
Jun 15 05:55:39.799935 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.799914    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:39.804793 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.804767    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jun 15 05:55:39.804793 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.804784    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jun 15 05:55:39.805785 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.805767    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6gghq\""
Jun 15 05:55:39.816649 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.816628    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"]
Jun 15 05:55:39.904014 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.903977    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:39.904183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.904025    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bchsj\" (UniqueName: \"kubernetes.io/projected/541aa8d6-c347-4d79-a3ff-68f49872f300-kube-api-access-bchsj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:39.904183 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:39.904080    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:40.005036 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:40.004993    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:40.005036 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:40.005045    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:40.005234 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:40.005088    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bchsj\" (UniqueName: \"kubernetes.io/projected/541aa8d6-c347-4d79-a3ff-68f49872f300-kube-api-access-bchsj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:40.005404 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:40.005361    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-bundle\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:40.005496 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:40.005476    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-util\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:40.018993 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:40.018969    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bchsj\" (UniqueName: \"kubernetes.io/projected/541aa8d6-c347-4d79-a3ff-68f49872f300-kube-api-access-bchsj\") pod \"59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") " pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:40.108765 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:40.108680    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:40.249555 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:40.249521    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"]
Jun 15 05:55:40.252716 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:55:40.252680    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541aa8d6_c347_4d79_a3ff_68f49872f300.slice/crio-6730fd88689493b7538349379711e94febdf8902d5d77b580597481d0337937b WatchSource:0}: Error finding container 6730fd88689493b7538349379711e94febdf8902d5d77b580597481d0337937b: Status 404 returned error can't find the container with id 6730fd88689493b7538349379711e94febdf8902d5d77b580597481d0337937b
Jun 15 05:55:41.125055 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:41.125016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl" event={"ID":"541aa8d6-c347-4d79-a3ff-68f49872f300","Type":"ContainerStarted","Data":"6730fd88689493b7538349379711e94febdf8902d5d77b580597481d0337937b"}
Jun 15 05:55:46.147511 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:46.147421    2562 generic.go:356] "Generic (PLEG): container finished" podID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerID="ddc1742cb2b7b25d66c33b95d33d302fac2b8f51263b05b808fa8448d0231a96" exitCode=0
Jun 15 05:55:46.147511 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:46.147490    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl" event={"ID":"541aa8d6-c347-4d79-a3ff-68f49872f300","Type":"ContainerDied","Data":"ddc1742cb2b7b25d66c33b95d33d302fac2b8f51263b05b808fa8448d0231a96"}
Jun 15 05:55:48.156083 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:48.156043    2562 generic.go:356] "Generic (PLEG): container finished" podID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerID="ef4813ea23b84a21eca95f917d2566c6cb18197ece6e049d030f56f9ba4e6575" exitCode=0
Jun 15 05:55:48.156478 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:48.156127    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl" event={"ID":"541aa8d6-c347-4d79-a3ff-68f49872f300","Type":"ContainerDied","Data":"ef4813ea23b84a21eca95f917d2566c6cb18197ece6e049d030f56f9ba4e6575"}
Jun 15 05:55:56.189222 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:56.189181    2562 generic.go:356] "Generic (PLEG): container finished" podID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerID="533ed478c76423de494c0bec434cf54b4f77916b39374904ea44d922f9c53085" exitCode=0
Jun 15 05:55:56.189634 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:56.189228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl" event={"ID":"541aa8d6-c347-4d79-a3ff-68f49872f300","Type":"ContainerDied","Data":"533ed478c76423de494c0bec434cf54b4f77916b39374904ea44d922f9c53085"}
Jun 15 05:55:57.317527 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.317503    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:57.449599 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.449517    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bchsj\" (UniqueName: \"kubernetes.io/projected/541aa8d6-c347-4d79-a3ff-68f49872f300-kube-api-access-bchsj\") pod \"541aa8d6-c347-4d79-a3ff-68f49872f300\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") "
Jun 15 05:55:57.449599 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.449565    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-bundle\") pod \"541aa8d6-c347-4d79-a3ff-68f49872f300\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") "
Jun 15 05:55:57.449781 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.449603    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-util\") pod \"541aa8d6-c347-4d79-a3ff-68f49872f300\" (UID: \"541aa8d6-c347-4d79-a3ff-68f49872f300\") "
Jun 15 05:55:57.450150 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.450101    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-bundle" (OuterVolumeSpecName: "bundle") pod "541aa8d6-c347-4d79-a3ff-68f49872f300" (UID: "541aa8d6-c347-4d79-a3ff-68f49872f300"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:55:57.451865 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.451842    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541aa8d6-c347-4d79-a3ff-68f49872f300-kube-api-access-bchsj" (OuterVolumeSpecName: "kube-api-access-bchsj") pod "541aa8d6-c347-4d79-a3ff-68f49872f300" (UID: "541aa8d6-c347-4d79-a3ff-68f49872f300"). InnerVolumeSpecName "kube-api-access-bchsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:55:57.454732 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.454714    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-util" (OuterVolumeSpecName: "util") pod "541aa8d6-c347-4d79-a3ff-68f49872f300" (UID: "541aa8d6-c347-4d79-a3ff-68f49872f300"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:55:57.550430 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.550375    2562 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-util\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:57.550430 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.550425    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bchsj\" (UniqueName: \"kubernetes.io/projected/541aa8d6-c347-4d79-a3ff-68f49872f300-kube-api-access-bchsj\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:57.550430 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:57.550436    2562 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/541aa8d6-c347-4d79-a3ff-68f49872f300-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:55:58.197981 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:58.197948    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl" event={"ID":"541aa8d6-c347-4d79-a3ff-68f49872f300","Type":"ContainerDied","Data":"6730fd88689493b7538349379711e94febdf8902d5d77b580597481d0337937b"}
Jun 15 05:55:58.197981 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:58.197983    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/59039e319e11338a40c6b6f1054d265f40bb50ceac6068d5c59955d29c7gfkl"
Jun 15 05:55:58.198179 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:55:58.197994    2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6730fd88689493b7538349379711e94febdf8902d5d77b580597481d0337937b"
Jun 15 05:56:01.384880 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.384849    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"]
Jun 15 05:56:01.385256 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.385100    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerName="pull"
Jun 15 05:56:01.385256 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.385111    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerName="pull"
Jun 15 05:56:01.385256 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.385121    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerName="util"
Jun 15 05:56:01.385256 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.385127    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerName="util"
Jun 15 05:56:01.385256 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.385138    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerName="extract"
Jun 15 05:56:01.385256 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.385143    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerName="extract"
Jun 15 05:56:01.385256 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.385188    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="541aa8d6-c347-4d79-a3ff-68f49872f300" containerName="extract"
Jun 15 05:56:01.428840 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.428808    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"]
Jun 15 05:56:01.428996 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.428908    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:01.431923 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.431894    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jun 15 05:56:01.432078 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.431912    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jun 15 05:56:01.432243 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.432197    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jun 15 05:56:01.432475 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.432459    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-p7q7g\""
Jun 15 05:56:01.481472 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.481437    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvjlc\" (UniqueName: \"kubernetes.io/projected/ae5b2a7d-0b7e-4257-827d-a104c900c6e7-kube-api-access-gvjlc\") pod \"custom-metrics-autoscaler-operator-6685d4df84-6lcsw\" (UID: \"ae5b2a7d-0b7e-4257-827d-a104c900c6e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:01.481655 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.481562    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ae5b2a7d-0b7e-4257-827d-a104c900c6e7-certificates\") pod \"custom-metrics-autoscaler-operator-6685d4df84-6lcsw\" (UID: \"ae5b2a7d-0b7e-4257-827d-a104c900c6e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:01.582568 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.582535    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjlc\" (UniqueName: \"kubernetes.io/projected/ae5b2a7d-0b7e-4257-827d-a104c900c6e7-kube-api-access-gvjlc\") pod \"custom-metrics-autoscaler-operator-6685d4df84-6lcsw\" (UID: \"ae5b2a7d-0b7e-4257-827d-a104c900c6e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:01.582765 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.582742    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ae5b2a7d-0b7e-4257-827d-a104c900c6e7-certificates\") pod \"custom-metrics-autoscaler-operator-6685d4df84-6lcsw\" (UID: \"ae5b2a7d-0b7e-4257-827d-a104c900c6e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:01.585226 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.585205    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ae5b2a7d-0b7e-4257-827d-a104c900c6e7-certificates\") pod \"custom-metrics-autoscaler-operator-6685d4df84-6lcsw\" (UID: \"ae5b2a7d-0b7e-4257-827d-a104c900c6e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:01.593247 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.593215    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvjlc\" (UniqueName: \"kubernetes.io/projected/ae5b2a7d-0b7e-4257-827d-a104c900c6e7-kube-api-access-gvjlc\") pod \"custom-metrics-autoscaler-operator-6685d4df84-6lcsw\" (UID: \"ae5b2a7d-0b7e-4257-827d-a104c900c6e7\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:01.739686 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.739648    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:01.938866 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:01.938842    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"]
Jun 15 05:56:01.941147 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:56:01.941119    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae5b2a7d_0b7e_4257_827d_a104c900c6e7.slice/crio-442eafcd82a251dfba2d6893ec534312565e7af33e10561fe0b3c9ebf76f42d1 WatchSource:0}: Error finding container 442eafcd82a251dfba2d6893ec534312565e7af33e10561fe0b3c9ebf76f42d1: Status 404 returned error can't find the container with id 442eafcd82a251dfba2d6893ec534312565e7af33e10561fe0b3c9ebf76f42d1
Jun 15 05:56:02.215488 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:02.215402    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw" event={"ID":"ae5b2a7d-0b7e-4257-827d-a104c900c6e7","Type":"ContainerStarted","Data":"442eafcd82a251dfba2d6893ec534312565e7af33e10561fe0b3c9ebf76f42d1"}
Jun 15 05:56:06.103367 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.103331    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-674595557b-twzcz"]
Jun 15 05:56:06.105408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.105369    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.108834 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.108810    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jun 15 05:56:06.109036 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.109014    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vxgsk\""
Jun 15 05:56:06.109036 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.109032    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jun 15 05:56:06.129399 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.129357    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-674595557b-twzcz"]
Jun 15 05:56:06.226389 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.226361    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48qsx\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-kube-api-access-48qsx\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.226556 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.226421    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.226556 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.226493    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/96acad43-5ba3-4563-ad9f-3dc322ef76af-cabundle0\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.234407 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.234367    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw" event={"ID":"ae5b2a7d-0b7e-4257-827d-a104c900c6e7","Type":"ContainerStarted","Data":"d06b9e91c57c589d1d3df09efa8ed91d31c484be7d2ca86702105d71e6d94f36"}
Jun 15 05:56:06.234539 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.234451    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:06.254024 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.253976    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw" podStartSLOduration=1.732263252 podStartE2EDuration="5.253961412s" totalImagesPullingTime="3.52169816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:06.252521988 +0000 UTC m=+509.205388934" watchObservedRunningTime="2026-06-15 05:56:06.253961412 +0000 UTC m=+509.206828356"
Jun 15 05:56:06.327426 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.327364    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.327426 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.327429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/96acad43-5ba3-4563-ad9f-3dc322ef76af-cabundle0\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.327680 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.327480    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48qsx\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-kube-api-access-48qsx\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.327680 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.327608    2562 secret.go:281] references non-existent secret key: ca.crt
Jun 15 05:56:06.327680 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.327630    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jun 15 05:56:06.327680 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.327642    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-674595557b-twzcz: references non-existent secret key: ca.crt
Jun 15 05:56:06.327844 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.327703    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates podName:96acad43-5ba3-4563-ad9f-3dc322ef76af nodeName:}" failed. No retries permitted until 2026-06-15 05:56:06.827682808 +0000 UTC m=+509.780549748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates") pod "keda-operator-674595557b-twzcz" (UID: "96acad43-5ba3-4563-ad9f-3dc322ef76af") : references non-existent secret key: ca.crt
Jun 15 05:56:06.328147 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.328129    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/96acad43-5ba3-4563-ad9f-3dc322ef76af-cabundle0\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.336808 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.336782    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48qsx\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-kube-api-access-48qsx\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.428868 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.428786    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"]
Jun 15 05:56:06.431294 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.431278    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.434011 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.433987    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jun 15 05:56:06.445852 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.445829    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"]
Jun 15 05:56:06.529812 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.529775    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c6197117-b1ab-4f92-8b25-12ae8de3e40b-temp-vol\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.529991 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.529822    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7xc\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-kube-api-access-9q7xc\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.529991 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.529854    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.631283 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.631253    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c6197117-b1ab-4f92-8b25-12ae8de3e40b-temp-vol\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.631485 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.631293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7xc\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-kube-api-access-9q7xc\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.631485 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.631313    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.631485 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.631434    2562 secret.go:281] references non-existent secret key: tls.crt
Jun 15 05:56:06.631485 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.631447    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jun 15 05:56:06.631485 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.631464    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk: references non-existent secret key: tls.crt
Jun 15 05:56:06.631799 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.631515    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates podName:c6197117-b1ab-4f92-8b25-12ae8de3e40b nodeName:}" failed. No retries permitted until 2026-06-15 05:56:07.1314989 +0000 UTC m=+510.084365824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates") pod "keda-metrics-apiserver-68788fbd9b-7qzkk" (UID: "c6197117-b1ab-4f92-8b25-12ae8de3e40b") : references non-existent secret key: tls.crt
Jun 15 05:56:06.631799 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.631733    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c6197117-b1ab-4f92-8b25-12ae8de3e40b-temp-vol\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.642114 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.642085    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7xc\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-kube-api-access-9q7xc\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:06.724783 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.724743    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-c6d879546-8lwkq"]
Jun 15 05:56:06.727005 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.726977    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:06.730671 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.730654    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jun 15 05:56:06.738811 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.738790    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-c6d879546-8lwkq"]
Jun 15 05:56:06.832328 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.832293    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:06.832540 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.832363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwvtq\" (UniqueName: \"kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-kube-api-access-gwvtq\") pod \"keda-admission-c6d879546-8lwkq\" (UID: \"a94beea4-3421-47ce-9920-29399ca5a114\") " pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:06.832540 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.832444    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-certificates\") pod \"keda-admission-c6d879546-8lwkq\" (UID: \"a94beea4-3421-47ce-9920-29399ca5a114\") " pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:06.832540 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.832479    2562 secret.go:281] references non-existent secret key: ca.crt
Jun 15 05:56:06.832540 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.832499    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jun 15 05:56:06.832540 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.832509    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-674595557b-twzcz: references non-existent secret key: ca.crt
Jun 15 05:56:06.832723 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.832568    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates podName:96acad43-5ba3-4563-ad9f-3dc322ef76af nodeName:}" failed. No retries permitted until 2026-06-15 05:56:07.832550572 +0000 UTC m=+510.785417497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates") pod "keda-operator-674595557b-twzcz" (UID: "96acad43-5ba3-4563-ad9f-3dc322ef76af") : references non-existent secret key: ca.crt
Jun 15 05:56:06.933863 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.933820    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-certificates\") pod \"keda-admission-c6d879546-8lwkq\" (UID: \"a94beea4-3421-47ce-9920-29399ca5a114\") " pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:06.934051 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.933960    2562 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jun 15 05:56:06.934051 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.933966    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwvtq\" (UniqueName: \"kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-kube-api-access-gwvtq\") pod \"keda-admission-c6d879546-8lwkq\" (UID: \"a94beea4-3421-47ce-9920-29399ca5a114\") " pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:06.934051 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.933980    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-c6d879546-8lwkq: secret "keda-admission-webhooks-certs" not found
Jun 15 05:56:06.934173 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:06.934074    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-certificates podName:a94beea4-3421-47ce-9920-29399ca5a114 nodeName:}" failed. No retries permitted until 2026-06-15 05:56:07.43405985 +0000 UTC m=+510.386926772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-certificates") pod "keda-admission-c6d879546-8lwkq" (UID: "a94beea4-3421-47ce-9920-29399ca5a114") : secret "keda-admission-webhooks-certs" not found
Jun 15 05:56:06.950424 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:06.950371    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwvtq\" (UniqueName: \"kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-kube-api-access-gwvtq\") pod \"keda-admission-c6d879546-8lwkq\" (UID: \"a94beea4-3421-47ce-9920-29399ca5a114\") " pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:07.135938 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:07.135842    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:07.136329 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:07.135998    2562 secret.go:281] references non-existent secret key: tls.crt
Jun 15 05:56:07.136329 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:07.136020    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jun 15 05:56:07.136329 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:07.136038    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk: references non-existent secret key: tls.crt
Jun 15 05:56:07.136329 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:07.136090    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates podName:c6197117-b1ab-4f92-8b25-12ae8de3e40b nodeName:}" failed. No retries permitted until 2026-06-15 05:56:08.136074438 +0000 UTC m=+511.088941361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates") pod "keda-metrics-apiserver-68788fbd9b-7qzkk" (UID: "c6197117-b1ab-4f92-8b25-12ae8de3e40b") : references non-existent secret key: tls.crt
Jun 15 05:56:07.439310 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:07.439224    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-certificates\") pod \"keda-admission-c6d879546-8lwkq\" (UID: \"a94beea4-3421-47ce-9920-29399ca5a114\") " pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:07.441897 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:07.441875    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a94beea4-3421-47ce-9920-29399ca5a114-certificates\") pod \"keda-admission-c6d879546-8lwkq\" (UID: \"a94beea4-3421-47ce-9920-29399ca5a114\") " pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:07.639893 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:07.639845    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:07.776568 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:07.776533    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-c6d879546-8lwkq"]
Jun 15 05:56:07.780543 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:56:07.780512    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94beea4_3421_47ce_9920_29399ca5a114.slice/crio-c3870594da77dc7d3a0bd92f258c9a67285503b8e09abfcf81d3e6a7fcc9c816 WatchSource:0}: Error finding container c3870594da77dc7d3a0bd92f258c9a67285503b8e09abfcf81d3e6a7fcc9c816: Status 404 returned error can't find the container with id c3870594da77dc7d3a0bd92f258c9a67285503b8e09abfcf81d3e6a7fcc9c816
Jun 15 05:56:07.844158 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:07.844120    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:07.844334 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:07.844272    2562 secret.go:281] references non-existent secret key: ca.crt
Jun 15 05:56:07.844334 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:07.844293    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jun 15 05:56:07.844334 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:07.844302    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-674595557b-twzcz: references non-existent secret key: ca.crt
Jun 15 05:56:07.844481 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:07.844354    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates podName:96acad43-5ba3-4563-ad9f-3dc322ef76af nodeName:}" failed. No retries permitted until 2026-06-15 05:56:09.844338243 +0000 UTC m=+512.797205166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates") pod "keda-operator-674595557b-twzcz" (UID: "96acad43-5ba3-4563-ad9f-3dc322ef76af") : references non-existent secret key: ca.crt
Jun 15 05:56:08.147548 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:08.147503    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:08.147916 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:08.147645    2562 secret.go:281] references non-existent secret key: tls.crt
Jun 15 05:56:08.147916 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:08.147664    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jun 15 05:56:08.147916 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:08.147681    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk: references non-existent secret key: tls.crt
Jun 15 05:56:08.147916 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:08.147729    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates podName:c6197117-b1ab-4f92-8b25-12ae8de3e40b nodeName:}" failed. No retries permitted until 2026-06-15 05:56:10.147716051 +0000 UTC m=+513.100582975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates") pod "keda-metrics-apiserver-68788fbd9b-7qzkk" (UID: "c6197117-b1ab-4f92-8b25-12ae8de3e40b") : references non-existent secret key: tls.crt
Jun 15 05:56:08.243370 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:08.243326    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-c6d879546-8lwkq" event={"ID":"a94beea4-3421-47ce-9920-29399ca5a114","Type":"ContainerStarted","Data":"c3870594da77dc7d3a0bd92f258c9a67285503b8e09abfcf81d3e6a7fcc9c816"}
Jun 15 05:56:09.248602 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:09.248567    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-c6d879546-8lwkq" event={"ID":"a94beea4-3421-47ce-9920-29399ca5a114","Type":"ContainerStarted","Data":"7ca94a3123002353876414a2406383bf999ff010f6b2e7382945771c994c550a"}
Jun 15 05:56:09.249055 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:09.248772    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:09.271836 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:09.271789    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-c6d879546-8lwkq" podStartSLOduration=2.021592647 podStartE2EDuration="3.271776062s" totalImagesPullingTime="1.250183415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:09.270447024 +0000 UTC m=+512.223313971" watchObservedRunningTime="2026-06-15 05:56:09.271776062 +0000 UTC m=+512.224643018"
Jun 15 05:56:09.862498 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:09.862463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:09.862676 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:09.862622    2562 secret.go:281] references non-existent secret key: ca.crt
Jun 15 05:56:09.862676 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:09.862646    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jun 15 05:56:09.862676 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:09.862654    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-674595557b-twzcz: references non-existent secret key: ca.crt
Jun 15 05:56:09.862778 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:09.862706    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates podName:96acad43-5ba3-4563-ad9f-3dc322ef76af nodeName:}" failed. No retries permitted until 2026-06-15 05:56:13.862692547 +0000 UTC m=+516.815559469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates") pod "keda-operator-674595557b-twzcz" (UID: "96acad43-5ba3-4563-ad9f-3dc322ef76af") : references non-existent secret key: ca.crt
Jun 15 05:56:10.165165 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:10.165074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:10.165312 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:10.165186    2562 secret.go:281] references non-existent secret key: tls.crt
Jun 15 05:56:10.165312 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:10.165198    2562 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jun 15 05:56:10.165312 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:10.165218    2562 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk: references non-existent secret key: tls.crt
Jun 15 05:56:10.165312 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:56:10.165276    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates podName:c6197117-b1ab-4f92-8b25-12ae8de3e40b nodeName:}" failed. No retries permitted until 2026-06-15 05:56:14.165263006 +0000 UTC m=+517.118129930 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates") pod "keda-metrics-apiserver-68788fbd9b-7qzkk" (UID: "c6197117-b1ab-4f92-8b25-12ae8de3e40b") : references non-existent secret key: tls.crt
Jun 15 05:56:13.896616 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:13.896576    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:13.899067 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:13.899047    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96acad43-5ba3-4563-ad9f-3dc322ef76af-certificates\") pod \"keda-operator-674595557b-twzcz\" (UID: \"96acad43-5ba3-4563-ad9f-3dc322ef76af\") " pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:13.917115 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:13.917081    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:14.051350 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:14.051325    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-674595557b-twzcz"]
Jun 15 05:56:14.053642 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:56:14.053609    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96acad43_5ba3_4563_ad9f_3dc322ef76af.slice/crio-bd3c7f7a936c370349074dbd39ce48041f94bfd992b25c99e2a808e8748a3663 WatchSource:0}: Error finding container bd3c7f7a936c370349074dbd39ce48041f94bfd992b25c99e2a808e8748a3663: Status 404 returned error can't find the container with id bd3c7f7a936c370349074dbd39ce48041f94bfd992b25c99e2a808e8748a3663
Jun 15 05:56:14.199781 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:14.199668    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:14.202344 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:14.202317    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c6197117-b1ab-4f92-8b25-12ae8de3e40b-certificates\") pod \"keda-metrics-apiserver-68788fbd9b-7qzkk\" (UID: \"c6197117-b1ab-4f92-8b25-12ae8de3e40b\") " pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:14.241340 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:14.241303    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:14.273585 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:14.272698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-674595557b-twzcz" event={"ID":"96acad43-5ba3-4563-ad9f-3dc322ef76af","Type":"ContainerStarted","Data":"bd3c7f7a936c370349074dbd39ce48041f94bfd992b25c99e2a808e8748a3663"}
Jun 15 05:56:14.372892 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:14.372858    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"]
Jun 15 05:56:14.376328 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:56:14.376294    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6197117_b1ab_4f92_8b25_12ae8de3e40b.slice/crio-5d270b3e28c0391163e9ef549a97af7910bc597aa5fe015b95f55d66d6ee26fb WatchSource:0}: Error finding container 5d270b3e28c0391163e9ef549a97af7910bc597aa5fe015b95f55d66d6ee26fb: Status 404 returned error can't find the container with id 5d270b3e28c0391163e9ef549a97af7910bc597aa5fe015b95f55d66d6ee26fb
Jun 15 05:56:15.279194 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:15.279154    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk" event={"ID":"c6197117-b1ab-4f92-8b25-12ae8de3e40b","Type":"ContainerStarted","Data":"5d270b3e28c0391163e9ef549a97af7910bc597aa5fe015b95f55d66d6ee26fb"}
Jun 15 05:56:18.293665 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:18.293628    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk" event={"ID":"c6197117-b1ab-4f92-8b25-12ae8de3e40b","Type":"ContainerStarted","Data":"509b43ae7bad4573619cc8d572f044e8cceacc2d56b7f662737e4862a2755c85"}
Jun 15 05:56:18.294168 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:18.293732    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:18.295125 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:18.295101    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-674595557b-twzcz" event={"ID":"96acad43-5ba3-4563-ad9f-3dc322ef76af","Type":"ContainerStarted","Data":"3d8c1de709d0903f50b950de0e568e36c78072e26e96825c5a53c7602b3d4bf5"}
Jun 15 05:56:18.295253 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:18.295235    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:56:18.331290 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:18.331240    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk" podStartSLOduration=9.016531891 podStartE2EDuration="12.331224292s" totalImagesPullingTime="3.314692401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:18.330454429 +0000 UTC m=+521.283321374" watchObservedRunningTime="2026-06-15 05:56:18.331224292 +0000 UTC m=+521.284091238"
Jun 15 05:56:18.370328 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:18.370267    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-674595557b-twzcz" podStartSLOduration=8.732749908 podStartE2EDuration="12.370248964s" totalImagesPullingTime="3.637499056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:18.367246988 +0000 UTC m=+521.320113935" watchObservedRunningTime="2026-06-15 05:56:18.370248964 +0000 UTC m=+521.323115909"
Jun 15 05:56:27.241471 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:27.241373    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6685d4df84-6lcsw"
Jun 15 05:56:29.304075 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:29.304042    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-68788fbd9b-7qzkk"
Jun 15 05:56:30.254966 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:30.254939    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-c6d879546-8lwkq"
Jun 15 05:56:39.301182 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:56:39.301154    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-674595557b-twzcz"
Jun 15 05:57:16.375797 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.375757    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-46thm"]
Jun 15 05:57:16.377940 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.377923    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:16.384891 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.384867    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jun 15 05:57:16.385000 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.384891    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 15 05:57:16.391541 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.391519    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 15 05:57:16.395761 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.395747    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-wnv52\""
Jun 15 05:57:16.401687 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.401659    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-46thm"]
Jun 15 05:57:16.485513 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.485481    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5n8j\" (UniqueName: \"kubernetes.io/projected/e19cfb7f-66bd-4066-816b-b2bccc7bde79-kube-api-access-d5n8j\") pod \"kserve-controller-manager-6c7654bd99-46thm\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") " pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:16.485702 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.485530    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert\") pod \"kserve-controller-manager-6c7654bd99-46thm\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") " pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:16.491233 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.491210    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-j6zn7"]
Jun 15 05:57:16.493175 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.493159    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:16.496699 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.496674    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jun 15 05:57:16.496979 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.496966    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-fb2sw\""
Jun 15 05:57:16.507741 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.507708    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-j6zn7"]
Jun 15 05:57:16.585894 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.585846    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn7ct\" (UniqueName: \"kubernetes.io/projected/c41efd46-aacd-446f-9067-99f9e8b2b1fc-kube-api-access-cn7ct\") pod \"seaweedfs-64568bcd49-j6zn7\" (UID: \"c41efd46-aacd-446f-9067-99f9e8b2b1fc\") " pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:16.586100 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.585910    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert\") pod \"kserve-controller-manager-6c7654bd99-46thm\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") " pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:16.586100 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:57:16.586009    2562 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jun 15 05:57:16.586100 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.586036    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c41efd46-aacd-446f-9067-99f9e8b2b1fc-data\") pod \"seaweedfs-64568bcd49-j6zn7\" (UID: \"c41efd46-aacd-446f-9067-99f9e8b2b1fc\") " pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:16.586100 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:57:16.586082    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert podName:e19cfb7f-66bd-4066-816b-b2bccc7bde79 nodeName:}" failed. No retries permitted until 2026-06-15 05:57:17.086059669 +0000 UTC m=+580.038926607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert") pod "kserve-controller-manager-6c7654bd99-46thm" (UID: "e19cfb7f-66bd-4066-816b-b2bccc7bde79") : secret "kserve-webhook-server-cert" not found
Jun 15 05:57:16.586313 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.586244    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5n8j\" (UniqueName: \"kubernetes.io/projected/e19cfb7f-66bd-4066-816b-b2bccc7bde79-kube-api-access-d5n8j\") pod \"kserve-controller-manager-6c7654bd99-46thm\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") " pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:16.601808 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.601783    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5n8j\" (UniqueName: \"kubernetes.io/projected/e19cfb7f-66bd-4066-816b-b2bccc7bde79-kube-api-access-d5n8j\") pod \"kserve-controller-manager-6c7654bd99-46thm\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") " pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:16.687202 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.687101    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7ct\" (UniqueName: \"kubernetes.io/projected/c41efd46-aacd-446f-9067-99f9e8b2b1fc-kube-api-access-cn7ct\") pod \"seaweedfs-64568bcd49-j6zn7\" (UID: \"c41efd46-aacd-446f-9067-99f9e8b2b1fc\") " pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:16.687202 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.687197    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c41efd46-aacd-446f-9067-99f9e8b2b1fc-data\") pod \"seaweedfs-64568bcd49-j6zn7\" (UID: \"c41efd46-aacd-446f-9067-99f9e8b2b1fc\") " pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:16.687619 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.687600    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c41efd46-aacd-446f-9067-99f9e8b2b1fc-data\") pod \"seaweedfs-64568bcd49-j6zn7\" (UID: \"c41efd46-aacd-446f-9067-99f9e8b2b1fc\") " pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:16.702332 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.702301    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7ct\" (UniqueName: \"kubernetes.io/projected/c41efd46-aacd-446f-9067-99f9e8b2b1fc-kube-api-access-cn7ct\") pod \"seaweedfs-64568bcd49-j6zn7\" (UID: \"c41efd46-aacd-446f-9067-99f9e8b2b1fc\") " pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:16.802675 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.802628    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:16.957213 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:16.957186    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-j6zn7"]
Jun 15 05:57:16.959512 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:57:16.959483    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41efd46_aacd_446f_9067_99f9e8b2b1fc.slice/crio-2f6bb63e2f1eb872ceaa1cca1320a73b236630c85faaf543bacbde9da7d55a7c WatchSource:0}: Error finding container 2f6bb63e2f1eb872ceaa1cca1320a73b236630c85faaf543bacbde9da7d55a7c: Status 404 returned error can't find the container with id 2f6bb63e2f1eb872ceaa1cca1320a73b236630c85faaf543bacbde9da7d55a7c
Jun 15 05:57:17.089917 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:17.089869    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert\") pod \"kserve-controller-manager-6c7654bd99-46thm\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") " pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:17.092426 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:17.092406    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert\") pod \"kserve-controller-manager-6c7654bd99-46thm\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") " pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:17.288053 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:17.288018    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:17.446507 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:17.446478    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-46thm"]
Jun 15 05:57:17.506483 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:57:17.506451    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode19cfb7f_66bd_4066_816b_b2bccc7bde79.slice/crio-55a81b1c5ee8fade6532a1943f4a07bf604a064dbc9178542e52dab9a3046a49 WatchSource:0}: Error finding container 55a81b1c5ee8fade6532a1943f4a07bf604a064dbc9178542e52dab9a3046a49: Status 404 returned error can't find the container with id 55a81b1c5ee8fade6532a1943f4a07bf604a064dbc9178542e52dab9a3046a49
Jun 15 05:57:17.538399 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:17.538305    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-46thm" event={"ID":"e19cfb7f-66bd-4066-816b-b2bccc7bde79","Type":"ContainerStarted","Data":"55a81b1c5ee8fade6532a1943f4a07bf604a064dbc9178542e52dab9a3046a49"}
Jun 15 05:57:17.539644 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:17.539614    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-j6zn7" event={"ID":"c41efd46-aacd-446f-9067-99f9e8b2b1fc","Type":"ContainerStarted","Data":"2f6bb63e2f1eb872ceaa1cca1320a73b236630c85faaf543bacbde9da7d55a7c"}
Jun 15 05:57:20.558473 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:20.558433    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-46thm" event={"ID":"e19cfb7f-66bd-4066-816b-b2bccc7bde79","Type":"ContainerStarted","Data":"76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7"}
Jun 15 05:57:20.558983 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:20.558625    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:20.587232 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:20.587145    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6c7654bd99-46thm" podStartSLOduration=2.10907883 podStartE2EDuration="4.587093387s" totalImagesPullingTime="2.478014557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:57:20.586540264 +0000 UTC m=+583.539407210" watchObservedRunningTime="2026-06-15 05:57:20.587093387 +0000 UTC m=+583.539960333"
Jun 15 05:57:22.568315 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:22.568276    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-j6zn7" event={"ID":"c41efd46-aacd-446f-9067-99f9e8b2b1fc","Type":"ContainerStarted","Data":"42cf95aaec7b5f7f482241f8538fb667edcaecfb2bdd88b6820e2a1a01a81a94"}
Jun 15 05:57:22.568774 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:22.568413    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:22.589537 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:22.589487    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-j6zn7" podStartSLOduration=1.92697119 podStartE2EDuration="6.589469741s" totalImagesPullingTime="4.662498551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:57:22.588733015 +0000 UTC m=+585.541599960" watchObservedRunningTime="2026-06-15 05:57:22.589469741 +0000 UTC m=+585.542336685"
Jun 15 05:57:28.575361 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:28.575326    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-j6zn7"
Jun 15 05:57:37.460876 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:37.460850    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:57:37.461295 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:37.460905    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 05:57:51.567770 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:51.567735    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:51.809531 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:51.809493    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-46thm"]
Jun 15 05:57:51.809719 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:51.809674    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6c7654bd99-46thm" podUID="e19cfb7f-66bd-4066-816b-b2bccc7bde79" containerName="manager" containerID="cri-o://76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7" gracePeriod=10
Jun 15 05:57:51.845600 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:51.845565    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-d26gb"]
Jun 15 05:57:51.847757 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:51.847740    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:51.858659 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:51.858632    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-d26gb"]
Jun 15 05:57:51.959964 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:51.959924    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkt8k\" (UniqueName: \"kubernetes.io/projected/360b7d2c-63c7-4cb1-b793-d0b7d942e733-kube-api-access-tkt8k\") pod \"kserve-controller-manager-6c7654bd99-d26gb\" (UID: \"360b7d2c-63c7-4cb1-b793-d0b7d942e733\") " pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:51.960141 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:51.960042    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/360b7d2c-63c7-4cb1-b793-d0b7d942e733-cert\") pod \"kserve-controller-manager-6c7654bd99-d26gb\" (UID: \"360b7d2c-63c7-4cb1-b793-d0b7d942e733\") " pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:52.054606 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.054581    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:52.060894 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.060871    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/360b7d2c-63c7-4cb1-b793-d0b7d942e733-cert\") pod \"kserve-controller-manager-6c7654bd99-d26gb\" (UID: \"360b7d2c-63c7-4cb1-b793-d0b7d942e733\") " pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:52.060957 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.060908    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkt8k\" (UniqueName: \"kubernetes.io/projected/360b7d2c-63c7-4cb1-b793-d0b7d942e733-kube-api-access-tkt8k\") pod \"kserve-controller-manager-6c7654bd99-d26gb\" (UID: \"360b7d2c-63c7-4cb1-b793-d0b7d942e733\") " pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:52.063350 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.063322    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/360b7d2c-63c7-4cb1-b793-d0b7d942e733-cert\") pod \"kserve-controller-manager-6c7654bd99-d26gb\" (UID: \"360b7d2c-63c7-4cb1-b793-d0b7d942e733\") " pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:52.070476 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.070447    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkt8k\" (UniqueName: \"kubernetes.io/projected/360b7d2c-63c7-4cb1-b793-d0b7d942e733-kube-api-access-tkt8k\") pod \"kserve-controller-manager-6c7654bd99-d26gb\" (UID: \"360b7d2c-63c7-4cb1-b793-d0b7d942e733\") " pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:52.161683 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.161589    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5n8j\" (UniqueName: \"kubernetes.io/projected/e19cfb7f-66bd-4066-816b-b2bccc7bde79-kube-api-access-d5n8j\") pod \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") "
Jun 15 05:57:52.161683 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.161637    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert\") pod \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\" (UID: \"e19cfb7f-66bd-4066-816b-b2bccc7bde79\") "
Jun 15 05:57:52.163988 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.163957    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert" (OuterVolumeSpecName: "cert") pod "e19cfb7f-66bd-4066-816b-b2bccc7bde79" (UID: "e19cfb7f-66bd-4066-816b-b2bccc7bde79"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:57:52.163988 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.163967    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e19cfb7f-66bd-4066-816b-b2bccc7bde79-kube-api-access-d5n8j" (OuterVolumeSpecName: "kube-api-access-d5n8j") pod "e19cfb7f-66bd-4066-816b-b2bccc7bde79" (UID: "e19cfb7f-66bd-4066-816b-b2bccc7bde79"). InnerVolumeSpecName "kube-api-access-d5n8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:57:52.184218 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.184180    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:52.262806 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.262759    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5n8j\" (UniqueName: \"kubernetes.io/projected/e19cfb7f-66bd-4066-816b-b2bccc7bde79-kube-api-access-d5n8j\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:57:52.262806 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.262793    2562 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e19cfb7f-66bd-4066-816b-b2bccc7bde79-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:57:52.317489 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.317461    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-d26gb"]
Jun 15 05:57:52.320007 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:57:52.319978    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod360b7d2c_63c7_4cb1_b793_d0b7d942e733.slice/crio-30d81f18c09145cd20f41e28c6f549dcc94407d38aa2100f18197a860238fbe5 WatchSource:0}: Error finding container 30d81f18c09145cd20f41e28c6f549dcc94407d38aa2100f18197a860238fbe5: Status 404 returned error can't find the container with id 30d81f18c09145cd20f41e28c6f549dcc94407d38aa2100f18197a860238fbe5
Jun 15 05:57:52.693895 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.693864    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-d26gb" event={"ID":"360b7d2c-63c7-4cb1-b793-d0b7d942e733","Type":"ContainerStarted","Data":"30d81f18c09145cd20f41e28c6f549dcc94407d38aa2100f18197a860238fbe5"}
Jun 15 05:57:52.695032 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.695010    2562 generic.go:356] "Generic (PLEG): container finished" podID="e19cfb7f-66bd-4066-816b-b2bccc7bde79" containerID="76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7" exitCode=0
Jun 15 05:57:52.695119 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.695050    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6c7654bd99-46thm"
Jun 15 05:57:52.695119 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.695050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-46thm" event={"ID":"e19cfb7f-66bd-4066-816b-b2bccc7bde79","Type":"ContainerDied","Data":"76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7"}
Jun 15 05:57:52.695119 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.695114    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-46thm" event={"ID":"e19cfb7f-66bd-4066-816b-b2bccc7bde79","Type":"ContainerDied","Data":"55a81b1c5ee8fade6532a1943f4a07bf604a064dbc9178542e52dab9a3046a49"}
Jun 15 05:57:52.695215 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.695147    2562 scope.go:117] "RemoveContainer" containerID="76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7"
Jun 15 05:57:52.704352 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.704320    2562 scope.go:117] "RemoveContainer" containerID="76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7"
Jun 15 05:57:52.704724 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:57:52.704705    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7\": container with ID starting with 76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7 not found: ID does not exist" containerID="76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7"
Jun 15 05:57:52.704798 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.704731    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7"} err="failed to get container status \"76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7\": rpc error: code = NotFound desc = could not find container \"76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7\": container with ID starting with 76021fc08d07b2d09716e506c81efee2e6d01a343c562058d16e10f4e07216d7 not found: ID does not exist"
Jun 15 05:57:52.717726 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.717701    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-46thm"]
Jun 15 05:57:52.722097 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:52.722069    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6c7654bd99-46thm"]
Jun 15 05:57:53.489882 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:53.489848    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e19cfb7f-66bd-4066-816b-b2bccc7bde79" path="/var/lib/kubelet/pods/e19cfb7f-66bd-4066-816b-b2bccc7bde79/volumes"
Jun 15 05:57:53.700128 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:53.700097    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6c7654bd99-d26gb" event={"ID":"360b7d2c-63c7-4cb1-b793-d0b7d942e733","Type":"ContainerStarted","Data":"4a5c0394df72ba133d1305c818d44e7fa1f1438f6c6f9393627adafc79377f03"}
Jun 15 05:57:53.700606 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:53.700200    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:57:53.716883 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:57:53.716833    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6c7654bd99-d26gb" podStartSLOduration=2.347941234 podStartE2EDuration="2.7168168s" totalImagesPullingTime="368.875566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:57:53.716322484 +0000 UTC m=+616.669189454" watchObservedRunningTime="2026-06-15 05:57:53.7168168 +0000 UTC m=+616.669683746"
Jun 15 05:58:24.709519 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:24.709451    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6c7654bd99-d26gb"
Jun 15 05:58:25.622021 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.621988    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-n79ql"]
Jun 15 05:58:25.622302 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.622288    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e19cfb7f-66bd-4066-816b-b2bccc7bde79" containerName="manager"
Jun 15 05:58:25.622302 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.622303    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e19cfb7f-66bd-4066-816b-b2bccc7bde79" containerName="manager"
Jun 15 05:58:25.622400 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.622353    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e19cfb7f-66bd-4066-816b-b2bccc7bde79" containerName="manager"
Jun 15 05:58:25.625399 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.625370    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:25.628225 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.628201    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-hd2h9\""
Jun 15 05:58:25.629048 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.629034    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jun 15 05:58:25.638331 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.638307    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-n79ql"]
Jun 15 05:58:25.735062 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.735023    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2c4\" (UniqueName: \"kubernetes.io/projected/85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5-kube-api-access-pv2c4\") pod \"model-serving-api-fd65d7d6-n79ql\" (UID: \"85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5\") " pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:25.735477 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.735072    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5-tls-certs\") pod \"model-serving-api-fd65d7d6-n79ql\" (UID: \"85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5\") " pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:25.836310 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.836281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2c4\" (UniqueName: \"kubernetes.io/projected/85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5-kube-api-access-pv2c4\") pod \"model-serving-api-fd65d7d6-n79ql\" (UID: \"85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5\") " pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:25.836512 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.836326    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5-tls-certs\") pod \"model-serving-api-fd65d7d6-n79ql\" (UID: \"85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5\") " pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:25.838894 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.838859    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5-tls-certs\") pod \"model-serving-api-fd65d7d6-n79ql\" (UID: \"85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5\") " pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:25.848569 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.848544    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2c4\" (UniqueName: \"kubernetes.io/projected/85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5-kube-api-access-pv2c4\") pod \"model-serving-api-fd65d7d6-n79ql\" (UID: \"85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5\") " pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:25.935943 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:25.935856    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:26.070466 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:26.070400    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-n79ql"]
Jun 15 05:58:26.072766 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:58:26.072740    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85dfd84e_d031_49e5_8f5e_e89e4cdc2ef5.slice/crio-99f350aa79c00760c83bfb0445a987994b0660100ff35f0fa926665cb43f6848 WatchSource:0}: Error finding container 99f350aa79c00760c83bfb0445a987994b0660100ff35f0fa926665cb43f6848: Status 404 returned error can't find the container with id 99f350aa79c00760c83bfb0445a987994b0660100ff35f0fa926665cb43f6848
Jun 15 05:58:26.838903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:26.838866    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-n79ql" event={"ID":"85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5","Type":"ContainerStarted","Data":"99f350aa79c00760c83bfb0445a987994b0660100ff35f0fa926665cb43f6848"}
Jun 15 05:58:27.846840 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:27.846805    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-n79ql" event={"ID":"85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5","Type":"ContainerStarted","Data":"9a77fcd9ff681b399b8093d86597665d76150ce4000104be11e03fef4ffcf243"}
Jun 15 05:58:27.847238 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:27.846919    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:27.867413 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:27.867336    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-n79ql" podStartSLOduration=1.293176759 podStartE2EDuration="2.867319373s" totalImagesPullingTime="1.574142614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:27.866149935 +0000 UTC m=+650.819016881" watchObservedRunningTime="2026-06-15 05:58:27.867319373 +0000 UTC m=+650.820186434"
Jun 15 05:58:38.855304 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:38.855273    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-n79ql"
Jun 15 05:58:51.939223 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:51.939186    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bfc59d58d-txdlv"]
Jun 15 05:58:51.943887 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:51.943866    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:51.966140 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:51.966113    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bfc59d58d-txdlv"]
Jun 15 05:58:52.041958 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.041922    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-trusted-ca-bundle\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.041958 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.041954    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkp8f\" (UniqueName: \"kubernetes.io/projected/683f9fef-7c97-4f97-8f68-d8ece1476cdc-kube-api-access-dkp8f\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.042196 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.041982    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-serving-cert\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.042196 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.042074    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-config\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.042278 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.042235    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-oauth-config\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.042278 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.042259    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-service-ca\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.042352 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.042293    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-oauth-serving-cert\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.143516 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.143481    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-oauth-config\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.143516 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.143516    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-service-ca\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.143773 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.143543    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-oauth-serving-cert\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.143773 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.143578    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-trusted-ca-bundle\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.143773 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.143593    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkp8f\" (UniqueName: \"kubernetes.io/projected/683f9fef-7c97-4f97-8f68-d8ece1476cdc-kube-api-access-dkp8f\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.143773 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.143619    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-serving-cert\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.143773 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.143660    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-config\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.144484 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.144457    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-oauth-serving-cert\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.144597 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.144496    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-config\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.144597 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.144549    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-service-ca\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.144883 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.144859    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/683f9fef-7c97-4f97-8f68-d8ece1476cdc-trusted-ca-bundle\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.146290 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.146269    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-serving-cert\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.146413 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.146321    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/683f9fef-7c97-4f97-8f68-d8ece1476cdc-console-oauth-config\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.153759 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.153721    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkp8f\" (UniqueName: \"kubernetes.io/projected/683f9fef-7c97-4f97-8f68-d8ece1476cdc-kube-api-access-dkp8f\") pod \"console-7bfc59d58d-txdlv\" (UID: \"683f9fef-7c97-4f97-8f68-d8ece1476cdc\") " pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.253553 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.253518    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:58:52.429977 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.429954    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bfc59d58d-txdlv"]
Jun 15 05:58:52.432039 ip-10-0-133-111 kubenswrapper[2562]: W0615 05:58:52.432012    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod683f9fef_7c97_4f97_8f68_d8ece1476cdc.slice/crio-28dae3ab71a7780378809aba6698d2ba2a79fc4cb238d3e871bdef6fd1ad1105 WatchSource:0}: Error finding container 28dae3ab71a7780378809aba6698d2ba2a79fc4cb238d3e871bdef6fd1ad1105: Status 404 returned error can't find the container with id 28dae3ab71a7780378809aba6698d2ba2a79fc4cb238d3e871bdef6fd1ad1105
Jun 15 05:58:52.947844 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.947809    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfc59d58d-txdlv" event={"ID":"683f9fef-7c97-4f97-8f68-d8ece1476cdc","Type":"ContainerStarted","Data":"d67a517792cd9e4d2e7589d9ab5f63b53d85d118d09812ec4ceb756a1e223a51"}
Jun 15 05:58:52.948248 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.947862    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bfc59d58d-txdlv" event={"ID":"683f9fef-7c97-4f97-8f68-d8ece1476cdc","Type":"ContainerStarted","Data":"28dae3ab71a7780378809aba6698d2ba2a79fc4cb238d3e871bdef6fd1ad1105"}
Jun 15 05:58:52.969148 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:58:52.969094    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bfc59d58d-txdlv" podStartSLOduration=1.969079204 podStartE2EDuration="1.969079204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:52.969030455 +0000 UTC m=+675.921897401" watchObservedRunningTime="2026-06-15 05:58:52.969079204 +0000 UTC m=+675.921946150"
Jun 15 05:59:02.254603 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:02.254563    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:59:02.255097 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:02.254661    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:59:02.260428 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:02.260374    2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:59:02.994620 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:02.994593    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bfc59d58d-txdlv"
Jun 15 05:59:03.045370 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:03.045334    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7856cd94c7-8jd9k"]
Jun 15 05:59:28.072448 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.072360    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7856cd94c7-8jd9k" podUID="c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" containerName="console" containerID="cri-o://f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd" gracePeriod=15
Jun 15 05:59:28.318807 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.318781    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:59:28.448423 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448300    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-serving-cert\") pod \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") "
Jun 15 05:59:28.448423 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448348    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-service-ca\") pod \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") "
Jun 15 05:59:28.448423 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448401    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-oauth-serving-cert\") pod \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") "
Jun 15 05:59:28.448423 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448423    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45qch\" (UniqueName: \"kubernetes.io/projected/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-kube-api-access-45qch\") pod \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") "
Jun 15 05:59:28.448760 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448575    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-trusted-ca-bundle\") pod \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") "
Jun 15 05:59:28.448760 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448614    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-oauth-config\") pod \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") "
Jun 15 05:59:28.448760 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448667    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-config\") pod \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\" (UID: \"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40\") "
Jun 15 05:59:28.448917 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448851    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" (UID: "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:59:28.448917 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448860    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-service-ca" (OuterVolumeSpecName: "service-ca") pod "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" (UID: "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:59:28.449016 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448952    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" (UID: "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:59:28.449016 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.448975    2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-service-ca\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:59:28.449285 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.449021    2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-oauth-serving-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:59:28.449285 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.449189    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-config" (OuterVolumeSpecName: "console-config") pod "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" (UID: "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:59:28.450769 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.450741    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-kube-api-access-45qch" (OuterVolumeSpecName: "kube-api-access-45qch") pod "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" (UID: "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40"). InnerVolumeSpecName "kube-api-access-45qch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:59:28.450769 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.450763    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" (UID: "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:59:28.450924 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.450893    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" (UID: "c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:59:28.549903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.549872    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qch\" (UniqueName: \"kubernetes.io/projected/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-kube-api-access-45qch\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:59:28.549903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.549898    2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-trusted-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:59:28.549903 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.549907    2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-oauth-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:59:28.550107 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.549917    2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:59:28.550107 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:28.549928    2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40-console-serving-cert\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 05:59:29.097049 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.097013    2562 generic.go:356] "Generic (PLEG): container finished" podID="c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" containerID="f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd" exitCode=0
Jun 15 05:59:29.097573 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.097069    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7856cd94c7-8jd9k"
Jun 15 05:59:29.097573 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.097103    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7856cd94c7-8jd9k" event={"ID":"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40","Type":"ContainerDied","Data":"f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd"}
Jun 15 05:59:29.097573 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.097153    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7856cd94c7-8jd9k" event={"ID":"c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40","Type":"ContainerDied","Data":"7a78bcd9d1f0024aa38ffcc8c03ce978815078daade413db6bc59e519c2d46f5"}
Jun 15 05:59:29.097573 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.097174    2562 scope.go:117] "RemoveContainer" containerID="f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd"
Jun 15 05:59:29.106079 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.106059    2562 scope.go:117] "RemoveContainer" containerID="f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd"
Jun 15 05:59:29.106350 ip-10-0-133-111 kubenswrapper[2562]: E0615 05:59:29.106332    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd\": container with ID starting with f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd not found: ID does not exist" containerID="f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd"
Jun 15 05:59:29.106408 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.106360    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd"} err="failed to get container status \"f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd\": rpc error: code = NotFound desc = could not find container \"f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd\": container with ID starting with f31da2a3e57811cc90da92504e0033c869baa056df08ed171d3f08d55f785ddd not found: ID does not exist"
Jun 15 05:59:29.118179 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.118142    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7856cd94c7-8jd9k"]
Jun 15 05:59:29.123365 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.123338    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7856cd94c7-8jd9k"]
Jun 15 05:59:29.490198 ip-10-0-133-111 kubenswrapper[2562]: I0615 05:59:29.490167    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" path="/var/lib/kubelet/pods/c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40/volumes"
Jun 15 06:00:11.455752 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.455717    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"]
Jun 15 06:00:11.456951 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.456926    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" containerName="console"
Jun 15 06:00:11.456951 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.456950    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" containerName="console"
Jun 15 06:00:11.457114 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.457034    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a2e1cd-8b40-46e1-87fa-fb0bf4e4ca40" containerName="console"
Jun 15 06:00:11.460355 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.460332    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.462950 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.462926    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 15 06:00:11.463131 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.463114    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5cf27-predictor-serving-cert\""
Jun 15 06:00:11.463320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.463297    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 15 06:00:11.463469 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.463131    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5cf27-kube-rbac-proxy-sar-config\""
Jun 15 06:00:11.464025 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.464009    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tjxr4\""
Jun 15 06:00:11.468291 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.468269    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"]
Jun 15 06:00:11.496742 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.496714    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.496914 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.496746    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-proxy-tls\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.496914 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.496778    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jflx6\" (UniqueName: \"kubernetes.io/projected/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-api-access-jflx6\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.598012 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.597974    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.598012 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.598026    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-proxy-tls\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.598283 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.598138    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jflx6\" (UniqueName: \"kubernetes.io/projected/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-api-access-jflx6\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.598736 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.598715    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.600772 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.600746    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-proxy-tls\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.607793 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.607756    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jflx6\" (UniqueName: \"kubernetes.io/projected/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-api-access-jflx6\") pod \"success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.776105 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.776060    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:11.923272 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.923170    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"]
Jun 15 06:00:11.926154 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:00:11.926129    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd3d266_bbdf_4072_9450_37bc5ff2c4bd.slice/crio-8eaabfdc1b405dec87d756e8530a7849f90c43e20149a8fab078ea6a5f98b0ad WatchSource:0}: Error finding container 8eaabfdc1b405dec87d756e8530a7849f90c43e20149a8fab078ea6a5f98b0ad: Status 404 returned error can't find the container with id 8eaabfdc1b405dec87d756e8530a7849f90c43e20149a8fab078ea6a5f98b0ad
Jun 15 06:00:11.928492 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:11.928474    2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:00:12.279257 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:12.279222    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" event={"ID":"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd","Type":"ContainerStarted","Data":"8eaabfdc1b405dec87d756e8530a7849f90c43e20149a8fab078ea6a5f98b0ad"}
Jun 15 06:00:24.347828 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:24.347744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" event={"ID":"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd","Type":"ContainerStarted","Data":"861f6c30616861921f1108f7ec5886835f4355200855ec7a644bfee76201d224"}
Jun 15 06:00:26.357535 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:26.357502    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" event={"ID":"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd","Type":"ContainerStarted","Data":"6d899166bd3d77c4b1d7371ede2b6fcd62756cd4943e4e85cda4071ea8e3cbcf"}
Jun 15 06:00:26.357970 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:26.357565    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:26.357970 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:26.357727    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:26.358855 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:26.358831    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 15 06:00:26.377918 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:26.377866    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podStartSLOduration=1.739422626 podStartE2EDuration="15.377852943s" totalImagesPullingTime="13.638430317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:00:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:00:26.376431262 +0000 UTC m=+769.329298236" watchObservedRunningTime="2026-06-15 06:00:26.377852943 +0000 UTC m=+769.330719888"
Jun 15 06:00:27.362399 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:27.362348    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 15 06:00:32.366816 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:32.366784    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:00:32.367414 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:32.367370    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 15 06:00:42.368465 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:42.368361    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 15 06:00:52.368635 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:00:52.368584    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 15 06:01:02.368004 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:02.367906    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jun 15 06:01:12.368255 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:12.368224    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:01:31.276526 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.276487    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"]
Jun 15 06:01:31.278821 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.278805    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:31.281247 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.281224    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-5cf27-kube-rbac-proxy-sar-config\""
Jun 15 06:01:31.281365 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.281270    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-5cf27-serving-cert\""
Jun 15 06:01:31.289584 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.289559    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"]
Jun 15 06:01:31.372336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.372297    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls\") pod \"switch-graph-5cf27-599c8856d9-mrll6\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") " pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:31.372517 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.372363    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c92c82f-7aa6-468a-b3bc-d0a338a99334-openshift-service-ca-bundle\") pod \"switch-graph-5cf27-599c8856d9-mrll6\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") " pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:31.473316 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.473269    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls\") pod \"switch-graph-5cf27-599c8856d9-mrll6\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") " pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:31.473505 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.473345    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c92c82f-7aa6-468a-b3bc-d0a338a99334-openshift-service-ca-bundle\") pod \"switch-graph-5cf27-599c8856d9-mrll6\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") " pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:31.473505 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:01:31.473456    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-5cf27-serving-cert: secret "switch-graph-5cf27-serving-cert" not found
Jun 15 06:01:31.473583 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:01:31.473527    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls podName:5c92c82f-7aa6-468a-b3bc-d0a338a99334 nodeName:}" failed. No retries permitted until 2026-06-15 06:01:31.973510023 +0000 UTC m=+834.926376946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls") pod "switch-graph-5cf27-599c8856d9-mrll6" (UID: "5c92c82f-7aa6-468a-b3bc-d0a338a99334") : secret "switch-graph-5cf27-serving-cert" not found
Jun 15 06:01:31.473973 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.473952    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c92c82f-7aa6-468a-b3bc-d0a338a99334-openshift-service-ca-bundle\") pod \"switch-graph-5cf27-599c8856d9-mrll6\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") " pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:31.978543 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.978496    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls\") pod \"switch-graph-5cf27-599c8856d9-mrll6\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") " pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:31.981115 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:31.981093    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls\") pod \"switch-graph-5cf27-599c8856d9-mrll6\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") " pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:32.191930 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:32.191898    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:32.325136 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:32.325101    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"]
Jun 15 06:01:32.327660 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:01:32.327632    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c92c82f_7aa6_468a_b3bc_d0a338a99334.slice/crio-3aceb23d709562c7ffe3352af078edb2713ef9c9564dc10dfe68f530eed8edb9 WatchSource:0}: Error finding container 3aceb23d709562c7ffe3352af078edb2713ef9c9564dc10dfe68f530eed8edb9: Status 404 returned error can't find the container with id 3aceb23d709562c7ffe3352af078edb2713ef9c9564dc10dfe68f530eed8edb9
Jun 15 06:01:32.638124 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:32.638037    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" event={"ID":"5c92c82f-7aa6-468a-b3bc-d0a338a99334","Type":"ContainerStarted","Data":"3aceb23d709562c7ffe3352af078edb2713ef9c9564dc10dfe68f530eed8edb9"}
Jun 15 06:01:35.653415 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:35.653362    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" event={"ID":"5c92c82f-7aa6-468a-b3bc-d0a338a99334","Type":"ContainerStarted","Data":"e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983"}
Jun 15 06:01:35.653773 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:35.653438    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:35.670302 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:35.670256    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" podStartSLOduration=2.178479584 podStartE2EDuration="4.670239286s" totalImagesPullingTime="2.491759702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:01:35.669120853 +0000 UTC m=+838.621987799" watchObservedRunningTime="2026-06-15 06:01:35.670239286 +0000 UTC m=+838.623106231"
Jun 15 06:01:41.664069 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:41.664039    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:01:45.438111 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.438076    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"]
Jun 15 06:01:45.438597 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.438266    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27" containerID="cri-o://e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983" gracePeriod=30
Jun 15 06:01:45.567670 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.567633    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"]
Jun 15 06:01:45.567959 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.567932    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container" containerID="cri-o://861f6c30616861921f1108f7ec5886835f4355200855ec7a644bfee76201d224" gracePeriod=30
Jun 15 06:01:45.568039 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.567960    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kube-rbac-proxy" containerID="cri-o://6d899166bd3d77c4b1d7371ede2b6fcd62756cd4943e4e85cda4071ea8e3cbcf" gracePeriod=30
Jun 15 06:01:45.629206 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.629161    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"]
Jun 15 06:01:45.632661 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.632633    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.638539 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.638496    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ea4bd-kube-rbac-proxy-sar-config\""
Jun 15 06:01:45.638812 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.638772    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-ea4bd-predictor-serving-cert\""
Jun 15 06:01:45.651215 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.651006    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"]
Jun 15 06:01:45.696869 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.696795    2562 generic.go:356] "Generic (PLEG): container finished" podID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerID="6d899166bd3d77c4b1d7371ede2b6fcd62756cd4943e4e85cda4071ea8e3cbcf" exitCode=2
Jun 15 06:01:45.696996 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.696869    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" event={"ID":"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd","Type":"ContainerDied","Data":"6d899166bd3d77c4b1d7371ede2b6fcd62756cd4943e4e85cda4071ea8e3cbcf"}
Jun 15 06:01:45.791979 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.791940    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.792175 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.792051    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.792175 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.792076    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4jq5\" (UniqueName: \"kubernetes.io/projected/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-api-access-p4jq5\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.892739 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.892704    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.892739 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.892745    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4jq5\" (UniqueName: \"kubernetes.io/projected/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-api-access-p4jq5\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.893105 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.892790    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.893558 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.893534    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.895435 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.895414    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.901220 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.901195    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4jq5\" (UniqueName: \"kubernetes.io/projected/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-api-access-p4jq5\") pod \"success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") " pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:45.960954 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:45.960868    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:46.100273 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:46.100177    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"]
Jun 15 06:01:46.102872 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:01:46.102835    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e314429_bd2e_44b9_9c6e_f4fe2fbc0c82.slice/crio-eeaf437164db1c24dc303401672c8a15c386c45c2a204b46e5637ced93c2dc7d WatchSource:0}: Error finding container eeaf437164db1c24dc303401672c8a15c386c45c2a204b46e5637ced93c2dc7d: Status 404 returned error can't find the container with id eeaf437164db1c24dc303401672c8a15c386c45c2a204b46e5637ced93c2dc7d
Jun 15 06:01:46.662486 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:46.662443    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:46.703355 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:46.703322    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" event={"ID":"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82","Type":"ContainerStarted","Data":"40f12168b42aecb014ea4d4b1b08388d54c7551961c2f21936e61630af176ed2"}
Jun 15 06:01:46.703553 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:46.703395    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" event={"ID":"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82","Type":"ContainerStarted","Data":"c63fbcda38131d926e57e94e815fd1f7385e46114a409ebb9b5a9c17333f78f1"}
Jun 15 06:01:46.703553 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:46.703421    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" event={"ID":"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82","Type":"ContainerStarted","Data":"eeaf437164db1c24dc303401672c8a15c386c45c2a204b46e5637ced93c2dc7d"}
Jun 15 06:01:46.703553 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:46.703446    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:46.727996 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:46.727945    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podStartSLOduration=1.727929345 podStartE2EDuration="1.727929345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:01:46.725452307 +0000 UTC m=+849.678319254" watchObservedRunningTime="2026-06-15 06:01:46.727929345 +0000 UTC m=+849.680796289"
Jun 15 06:01:47.363358 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:47.363318    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jun 15 06:01:47.708212 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:47.708120    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:47.709557 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:47.709529    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jun 15 06:01:48.714616 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.714586    2562 generic.go:356] "Generic (PLEG): container finished" podID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerID="861f6c30616861921f1108f7ec5886835f4355200855ec7a644bfee76201d224" exitCode=0
Jun 15 06:01:48.715003 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.714672    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" event={"ID":"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd","Type":"ContainerDied","Data":"861f6c30616861921f1108f7ec5886835f4355200855ec7a644bfee76201d224"}
Jun 15 06:01:48.715081 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.715046    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jun 15 06:01:48.827086 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.827064    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:01:48.916926 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.916892    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jflx6\" (UniqueName: \"kubernetes.io/projected/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-api-access-jflx6\") pod \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") "
Jun 15 06:01:48.916926 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.916936    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-proxy-tls\") pod \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") "
Jun 15 06:01:48.917131 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.916976    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-rbac-proxy-sar-config\") pod \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\" (UID: \"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd\") "
Jun 15 06:01:48.917374 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.917343    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" (UID: "dbd3d266-bbdf-4072-9450-37bc5ff2c4bd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:01:48.919191 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.919139    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" (UID: "dbd3d266-bbdf-4072-9450-37bc5ff2c4bd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:01:48.919269 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:48.919255    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-api-access-jflx6" (OuterVolumeSpecName: "kube-api-access-jflx6") pod "dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" (UID: "dbd3d266-bbdf-4072-9450-37bc5ff2c4bd"). InnerVolumeSpecName "kube-api-access-jflx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:01:49.017820 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.017782    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jflx6\" (UniqueName: \"kubernetes.io/projected/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-api-access-jflx6\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:01:49.017820 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.017814    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:01:49.017820 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.017824    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:01:49.720426 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.720375    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4" event={"ID":"dbd3d266-bbdf-4072-9450-37bc5ff2c4bd","Type":"ContainerDied","Data":"8eaabfdc1b405dec87d756e8530a7849f90c43e20149a8fab078ea6a5f98b0ad"}
Jun 15 06:01:49.720906 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.720463    2562 scope.go:117] "RemoveContainer" containerID="6d899166bd3d77c4b1d7371ede2b6fcd62756cd4943e4e85cda4071ea8e3cbcf"
Jun 15 06:01:49.720906 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.720407    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"
Jun 15 06:01:49.728634 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.728616    2562 scope.go:117] "RemoveContainer" containerID="861f6c30616861921f1108f7ec5886835f4355200855ec7a644bfee76201d224"
Jun 15 06:01:49.738274 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.738246    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"]
Jun 15 06:01:49.745211 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:49.745188    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf27-predictor-99cc5fc54-gxmf4"]
Jun 15 06:01:51.489643 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:51.489609    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" path="/var/lib/kubelet/pods/dbd3d266-bbdf-4072-9450-37bc5ff2c4bd/volumes"
Jun 15 06:01:51.661854 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:51.661810    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:53.721288 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:53.721259    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:01:53.722012 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:53.721984    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jun 15 06:01:56.663046 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:56.663008    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:56.663517 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:01:56.663120    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:02:01.662280 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:01.662238    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:03.722659 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:03.722620    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jun 15 06:02:06.662598 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:06.662556    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:11.662824 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:11.662780    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:13.722991 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:13.722953    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jun 15 06:02:15.615136 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.615111    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:02:15.733145 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.733118    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c92c82f-7aa6-468a-b3bc-d0a338a99334-openshift-service-ca-bundle\") pod \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") "
Jun 15 06:02:15.733298 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.733177    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls\") pod \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\" (UID: \"5c92c82f-7aa6-468a-b3bc-d0a338a99334\") "
Jun 15 06:02:15.733512 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.733488    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c92c82f-7aa6-468a-b3bc-d0a338a99334-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5c92c82f-7aa6-468a-b3bc-d0a338a99334" (UID: "5c92c82f-7aa6-468a-b3bc-d0a338a99334"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:02:15.735264 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.735241    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c92c82f-7aa6-468a-b3bc-d0a338a99334" (UID: "5c92c82f-7aa6-468a-b3bc-d0a338a99334"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:02:15.834355 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.834317    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c92c82f-7aa6-468a-b3bc-d0a338a99334-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:02:15.834355 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.834350    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c92c82f-7aa6-468a-b3bc-d0a338a99334-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:02:15.835267 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.835235    2562 generic.go:356] "Generic (PLEG): container finished" podID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerID="e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983" exitCode=137
Jun 15 06:02:15.835366 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.835282    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" event={"ID":"5c92c82f-7aa6-468a-b3bc-d0a338a99334","Type":"ContainerDied","Data":"e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983"}
Jun 15 06:02:15.835366 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.835285    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"
Jun 15 06:02:15.835366 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.835320    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6" event={"ID":"5c92c82f-7aa6-468a-b3bc-d0a338a99334","Type":"ContainerDied","Data":"3aceb23d709562c7ffe3352af078edb2713ef9c9564dc10dfe68f530eed8edb9"}
Jun 15 06:02:15.835366 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.835341    2562 scope.go:117] "RemoveContainer" containerID="e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983"
Jun 15 06:02:15.844019 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.844004    2562 scope.go:117] "RemoveContainer" containerID="e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983"
Jun 15 06:02:15.844263 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:02:15.844245    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983\": container with ID starting with e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983 not found: ID does not exist" containerID="e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983"
Jun 15 06:02:15.844320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.844271    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983"} err="failed to get container status \"e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983\": rpc error: code = NotFound desc = could not find container \"e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983\": container with ID starting with e94f4d989afb276f9e08aaf01a0313454d1b2e31973059160b29820efc54c983 not found: ID does not exist"
Jun 15 06:02:15.857576 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.857549    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"]
Jun 15 06:02:15.862131 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:15.862111    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5cf27-599c8856d9-mrll6"]
Jun 15 06:02:17.489307 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:17.489272    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" path="/var/lib/kubelet/pods/5c92c82f-7aa6-468a-b3bc-d0a338a99334/volumes"
Jun 15 06:02:23.722332 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:23.722291    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jun 15 06:02:31.221951 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.221913    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"]
Jun 15 06:02:31.222336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222273    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27"
Jun 15 06:02:31.222336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222286    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27"
Jun 15 06:02:31.222336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222307    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container"
Jun 15 06:02:31.222336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222312    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container"
Jun 15 06:02:31.222336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222320    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kube-rbac-proxy"
Jun 15 06:02:31.222336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222326    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kube-rbac-proxy"
Jun 15 06:02:31.222564 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222402    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kserve-container"
Jun 15 06:02:31.222564 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222412    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd3d266-bbdf-4072-9450-37bc5ff2c4bd" containerName="kube-rbac-proxy"
Jun 15 06:02:31.222564 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.222417    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c92c82f-7aa6-468a-b3bc-d0a338a99334" containerName="switch-graph-5cf27"
Jun 15 06:02:31.225253 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.225235    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:31.227726 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.227705    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\""
Jun 15 06:02:31.227929 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.227902    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\""
Jun 15 06:02:31.234592 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.234570    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"]
Jun 15 06:02:31.366298 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.366263    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-openshift-service-ca-bundle\") pod \"model-chainer-567c7f9fdc-vssgs\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") " pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:31.366501 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.366334    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls\") pod \"model-chainer-567c7f9fdc-vssgs\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") " pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:31.467411 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.467349    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-openshift-service-ca-bundle\") pod \"model-chainer-567c7f9fdc-vssgs\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") " pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:31.467595 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.467463    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls\") pod \"model-chainer-567c7f9fdc-vssgs\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") " pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:31.467595 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:02:31.467581    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found
Jun 15 06:02:31.467735 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:02:31.467660    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls podName:a12e02e6-4b5d-499d-a8d1-3903df7bb8c8 nodeName:}" failed. No retries permitted until 2026-06-15 06:02:31.967637159 +0000 UTC m=+894.920504082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls") pod "model-chainer-567c7f9fdc-vssgs" (UID: "a12e02e6-4b5d-499d-a8d1-3903df7bb8c8") : secret "model-chainer-serving-cert" not found
Jun 15 06:02:31.468184 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.468154    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-openshift-service-ca-bundle\") pod \"model-chainer-567c7f9fdc-vssgs\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") " pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:31.972744 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.972707    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls\") pod \"model-chainer-567c7f9fdc-vssgs\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") " pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:31.975302 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:31.975282    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls\") pod \"model-chainer-567c7f9fdc-vssgs\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") " pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:32.136632 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:32.136584    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:32.272084 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:32.272036    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"]
Jun 15 06:02:32.274262 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:02:32.274234    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda12e02e6_4b5d_499d_a8d1_3903df7bb8c8.slice/crio-38359e7516fbb3705b5ed7a44e9d0f98ba6520766c8f17dd45865087ddf631e8 WatchSource:0}: Error finding container 38359e7516fbb3705b5ed7a44e9d0f98ba6520766c8f17dd45865087ddf631e8: Status 404 returned error can't find the container with id 38359e7516fbb3705b5ed7a44e9d0f98ba6520766c8f17dd45865087ddf631e8
Jun 15 06:02:32.911546 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:32.911515    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" event={"ID":"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8","Type":"ContainerStarted","Data":"28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4"}
Jun 15 06:02:32.911724 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:32.911563    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" event={"ID":"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8","Type":"ContainerStarted","Data":"38359e7516fbb3705b5ed7a44e9d0f98ba6520766c8f17dd45865087ddf631e8"}
Jun 15 06:02:32.911724 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:32.911596    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:32.930508 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:32.930460    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" podStartSLOduration=1.930442652 podStartE2EDuration="1.930442652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:02:32.928737017 +0000 UTC m=+895.881603961" watchObservedRunningTime="2026-06-15 06:02:32.930442652 +0000 UTC m=+895.883309594"
Jun 15 06:02:33.722590 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:33.722560    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:02:37.485679 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:37.485654    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:02:37.489047 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:37.489024    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:02:38.921163 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:38.921134    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:41.335350 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.335318    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"]
Jun 15 06:02:41.335864 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.335579    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer" containerID="cri-o://28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4" gracePeriod=30
Jun 15 06:02:41.499863 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.499823    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"]
Jun 15 06:02:41.505611 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.505584    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.508257 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.508230    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0ac5a-kube-rbac-proxy-sar-config\""
Jun 15 06:02:41.508571 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.508549    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0ac5a-predictor-serving-cert\""
Jun 15 06:02:41.516141 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.516114    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"]
Jun 15 06:02:41.657245 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.657155    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlcs5\" (UniqueName: \"kubernetes.io/projected/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-api-access-qlcs5\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.657245 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.657204    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.657457 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.657262    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5993ef3c-639e-43b7-9d21-d54e0c592262-proxy-tls\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.758496 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.758456    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlcs5\" (UniqueName: \"kubernetes.io/projected/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-api-access-qlcs5\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.758686 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.758520    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.758686 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.758554    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5993ef3c-639e-43b7-9d21-d54e0c592262-proxy-tls\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.759252 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.759232    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.761040 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.761023    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5993ef3c-639e-43b7-9d21-d54e0c592262-proxy-tls\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.766207 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.766188    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlcs5\" (UniqueName: \"kubernetes.io/projected/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-api-access-qlcs5\") pod \"success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") " pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.818161 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.818121    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:41.962145 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:41.962118    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"]
Jun 15 06:02:41.964327 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:02:41.964298    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5993ef3c_639e_43b7_9d21_d54e0c592262.slice/crio-29eacc795ed8f9c028d680ed70cc64dbd005b500fa03089a35c073b53241aede WatchSource:0}: Error finding container 29eacc795ed8f9c028d680ed70cc64dbd005b500fa03089a35c073b53241aede: Status 404 returned error can't find the container with id 29eacc795ed8f9c028d680ed70cc64dbd005b500fa03089a35c073b53241aede
Jun 15 06:02:42.959159 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:42.959122    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" event={"ID":"5993ef3c-639e-43b7-9d21-d54e0c592262","Type":"ContainerStarted","Data":"c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60"}
Jun 15 06:02:42.959631 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:42.959180    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" event={"ID":"5993ef3c-639e-43b7-9d21-d54e0c592262","Type":"ContainerStarted","Data":"fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16"}
Jun 15 06:02:42.959631 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:42.959201    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" event={"ID":"5993ef3c-639e-43b7-9d21-d54e0c592262","Type":"ContainerStarted","Data":"29eacc795ed8f9c028d680ed70cc64dbd005b500fa03089a35c073b53241aede"}
Jun 15 06:02:42.959631 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:42.959437    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:42.959631 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:42.959485    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:42.960730 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:42.960705    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jun 15 06:02:42.982882 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:42.982829    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podStartSLOduration=1.9828134469999998 podStartE2EDuration="1.982813447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:02:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:02:42.982627113 +0000 UTC m=+905.935494058" watchObservedRunningTime="2026-06-15 06:02:42.982813447 +0000 UTC m=+905.935680392"
Jun 15 06:02:43.920685 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:43.920638    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:43.963930 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:43.963886    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jun 15 06:02:45.674470 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.674433    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"]
Jun 15 06:02:45.677676 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.677660    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:45.680501 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.680477    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ea4bd-serving-cert\""
Jun 15 06:02:45.680867 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.680849    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-ea4bd-kube-rbac-proxy-sar-config\""
Jun 15 06:02:45.686867 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.686844    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc762f-ac90-4697-a69a-70447f4c4eeb-openshift-service-ca-bundle\") pod \"switch-graph-ea4bd-76d9f94847-lx78n\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") " pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:45.687014 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.686927    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls\") pod \"switch-graph-ea4bd-76d9f94847-lx78n\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") " pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:45.693344 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.693312    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"]
Jun 15 06:02:45.788084 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.788032    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls\") pod \"switch-graph-ea4bd-76d9f94847-lx78n\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") " pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:45.788084 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.788094    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc762f-ac90-4697-a69a-70447f4c4eeb-openshift-service-ca-bundle\") pod \"switch-graph-ea4bd-76d9f94847-lx78n\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") " pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:45.788436 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:02:45.788220    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-ea4bd-serving-cert: secret "switch-graph-ea4bd-serving-cert" not found
Jun 15 06:02:45.788436 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:02:45.788300    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls podName:8adc762f-ac90-4697-a69a-70447f4c4eeb nodeName:}" failed. No retries permitted until 2026-06-15 06:02:46.288276991 +0000 UTC m=+909.241143913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls") pod "switch-graph-ea4bd-76d9f94847-lx78n" (UID: "8adc762f-ac90-4697-a69a-70447f4c4eeb") : secret "switch-graph-ea4bd-serving-cert" not found
Jun 15 06:02:45.789034 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:45.789002    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc762f-ac90-4697-a69a-70447f4c4eeb-openshift-service-ca-bundle\") pod \"switch-graph-ea4bd-76d9f94847-lx78n\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") " pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:46.291920 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:46.291875    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls\") pod \"switch-graph-ea4bd-76d9f94847-lx78n\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") " pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:46.294523 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:46.294487    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls\") pod \"switch-graph-ea4bd-76d9f94847-lx78n\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") " pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:46.588297 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:46.588188    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:46.726281 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:46.726244    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"]
Jun 15 06:02:46.729252 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:02:46.729221    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adc762f_ac90_4697_a69a_70447f4c4eeb.slice/crio-86d1cffaaa3160dc401a4facc1389f9b6eb0e1c3d1db1035959c52cb3caa2fce WatchSource:0}: Error finding container 86d1cffaaa3160dc401a4facc1389f9b6eb0e1c3d1db1035959c52cb3caa2fce: Status 404 returned error can't find the container with id 86d1cffaaa3160dc401a4facc1389f9b6eb0e1c3d1db1035959c52cb3caa2fce
Jun 15 06:02:46.980512 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:46.980475    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" event={"ID":"8adc762f-ac90-4697-a69a-70447f4c4eeb","Type":"ContainerStarted","Data":"b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c"}
Jun 15 06:02:46.980701 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:46.980530    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" event={"ID":"8adc762f-ac90-4697-a69a-70447f4c4eeb","Type":"ContainerStarted","Data":"86d1cffaaa3160dc401a4facc1389f9b6eb0e1c3d1db1035959c52cb3caa2fce"}
Jun 15 06:02:46.980701 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:46.980612    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:47.003037 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:47.002987    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" podStartSLOduration=2.002970569 podStartE2EDuration="2.002970569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:02:47.002144253 +0000 UTC m=+909.955011211" watchObservedRunningTime="2026-06-15 06:02:47.002970569 +0000 UTC m=+909.955837514"
Jun 15 06:02:48.920245 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:48.920206    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:48.969905 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:48.969877    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:02:48.970442 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:48.970410    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jun 15 06:02:52.990754 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:52.990726    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:02:53.920684 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:53.920641    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:53.920900 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:53.920751    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:02:58.920525 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:58.920477    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:58.970912 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:02:58.970867    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jun 15 06:03:03.920315 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:03.920277    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:03:08.920739 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:08.920689    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:03:08.970596 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:08.970551    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jun 15 06:03:11.487680 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:11.487654    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:03:11.616011 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:11.615911    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-openshift-service-ca-bundle\") pod \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") "
Jun 15 06:03:11.616011 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:11.615970    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls\") pod \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\" (UID: \"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8\") "
Jun 15 06:03:11.616361 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:11.616328    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" (UID: "a12e02e6-4b5d-499d-a8d1-3903df7bb8c8"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:03:11.618192 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:11.618161    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" (UID: "a12e02e6-4b5d-499d-a8d1-3903df7bb8c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:03:11.716783 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:11.716749    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:03:11.716783 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:11.716777    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:03:12.093556 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.093518    2562 generic.go:356] "Generic (PLEG): container finished" podID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerID="28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4" exitCode=0
Jun 15 06:03:12.093728 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.093562    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"
Jun 15 06:03:12.093728 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.093597    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" event={"ID":"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8","Type":"ContainerDied","Data":"28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4"}
Jun 15 06:03:12.093728 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.093641    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs" event={"ID":"a12e02e6-4b5d-499d-a8d1-3903df7bb8c8","Type":"ContainerDied","Data":"38359e7516fbb3705b5ed7a44e9d0f98ba6520766c8f17dd45865087ddf631e8"}
Jun 15 06:03:12.093728 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.093663    2562 scope.go:117] "RemoveContainer" containerID="28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4"
Jun 15 06:03:12.102246 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.102230    2562 scope.go:117] "RemoveContainer" containerID="28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4"
Jun 15 06:03:12.102528 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:03:12.102510    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4\": container with ID starting with 28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4 not found: ID does not exist" containerID="28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4"
Jun 15 06:03:12.102589 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.102536    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4"} err="failed to get container status \"28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4\": rpc error: code = NotFound desc = could not find container \"28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4\": container with ID starting with 28bfb83322116e886b017ee40b700cd1cb85f7cf614d658f3c6d662d44e0f0c4 not found: ID does not exist"
Jun 15 06:03:12.114253 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.114228    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"]
Jun 15 06:03:12.120856 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:12.120835    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-567c7f9fdc-vssgs"]
Jun 15 06:03:13.489600 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:13.489568    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" path="/var/lib/kubelet/pods/a12e02e6-4b5d-499d-a8d1-3903df7bb8c8/volumes"
Jun 15 06:03:18.970618 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:18.970577    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jun 15 06:03:28.970535 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:28.970501    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:03:41.504515 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.504481    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"]
Jun 15 06:03:41.505039 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.505023    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer"
Jun 15 06:03:41.505091 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.505043    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer"
Jun 15 06:03:41.505157 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.505146    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a12e02e6-4b5d-499d-a8d1-3903df7bb8c8" containerName="model-chainer"
Jun 15 06:03:41.509456 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.509438    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:41.512348 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.512329    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-0ac5a-kube-rbac-proxy-sar-config\""
Jun 15 06:03:41.512463 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.512347    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-0ac5a-serving-cert\""
Jun 15 06:03:41.518867 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.518841    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"]
Jun 15 06:03:41.566913 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.566877    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls\") pod \"sequence-graph-0ac5a-665fcb9dd4-hxvgp\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") " pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:41.567083 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.566959    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f98403b-2ccc-48a2-a20c-806943d57b04-openshift-service-ca-bundle\") pod \"sequence-graph-0ac5a-665fcb9dd4-hxvgp\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") " pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:41.668220 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.668180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f98403b-2ccc-48a2-a20c-806943d57b04-openshift-service-ca-bundle\") pod \"sequence-graph-0ac5a-665fcb9dd4-hxvgp\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") " pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:41.668399 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.668268    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls\") pod \"sequence-graph-0ac5a-665fcb9dd4-hxvgp\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") " pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:41.668399 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:03:41.668371    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-0ac5a-serving-cert: secret "sequence-graph-0ac5a-serving-cert" not found
Jun 15 06:03:41.668488 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:03:41.668469    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls podName:6f98403b-2ccc-48a2-a20c-806943d57b04 nodeName:}" failed. No retries permitted until 2026-06-15 06:03:42.16844691 +0000 UTC m=+965.121313833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls") pod "sequence-graph-0ac5a-665fcb9dd4-hxvgp" (UID: "6f98403b-2ccc-48a2-a20c-806943d57b04") : secret "sequence-graph-0ac5a-serving-cert" not found
Jun 15 06:03:41.668948 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:41.668926    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f98403b-2ccc-48a2-a20c-806943d57b04-openshift-service-ca-bundle\") pod \"sequence-graph-0ac5a-665fcb9dd4-hxvgp\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") " pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:42.172468 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:42.172429    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls\") pod \"sequence-graph-0ac5a-665fcb9dd4-hxvgp\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") " pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:42.175016 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:42.174993    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls\") pod \"sequence-graph-0ac5a-665fcb9dd4-hxvgp\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") " pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:42.419754 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:42.419669    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:42.555079 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:42.555053    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"]
Jun 15 06:03:42.557490 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:03:42.557462    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f98403b_2ccc_48a2_a20c_806943d57b04.slice/crio-e188cb7d9a82c147301d863f77f178d18698e98f6152143ecdba79a9dd22ac8d WatchSource:0}: Error finding container e188cb7d9a82c147301d863f77f178d18698e98f6152143ecdba79a9dd22ac8d: Status 404 returned error can't find the container with id e188cb7d9a82c147301d863f77f178d18698e98f6152143ecdba79a9dd22ac8d
Jun 15 06:03:43.234992 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:43.234952    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" event={"ID":"6f98403b-2ccc-48a2-a20c-806943d57b04","Type":"ContainerStarted","Data":"7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c"}
Jun 15 06:03:43.235188 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:43.235016    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" event={"ID":"6f98403b-2ccc-48a2-a20c-806943d57b04","Type":"ContainerStarted","Data":"e188cb7d9a82c147301d863f77f178d18698e98f6152143ecdba79a9dd22ac8d"}
Jun 15 06:03:43.235188 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:43.235131    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:03:43.253743 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:43.253688    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" podStartSLOduration=2.253668454 podStartE2EDuration="2.253668454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:03:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:03:43.253238073 +0000 UTC m=+966.206105038" watchObservedRunningTime="2026-06-15 06:03:43.253668454 +0000 UTC m=+966.206535400"
Jun 15 06:03:49.244010 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:03:49.243984    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:07:37.516400 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:07:37.516356    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:07:37.519168 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:07:37.519148    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:11:00.477428 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.477369    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"]
Jun 15 06:11:00.477918 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.477612    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd" containerID="cri-o://b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c" gracePeriod=30
Jun 15 06:11:00.642010 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.641971    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"]
Jun 15 06:11:00.646903 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:00.646881    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-serving-cert: secret "success-200-isvc-ea4bd-predictor-serving-cert" not found
Jun 15 06:11:00.647023 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:00.646965    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls podName:1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82 nodeName:}" failed. No retries permitted until 2026-06-15 06:11:01.146944277 +0000 UTC m=+1404.099811207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls") pod "success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" (UID: "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82") : secret "success-200-isvc-ea4bd-predictor-serving-cert" not found
Jun 15 06:11:00.708740 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.708686    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"]
Jun 15 06:11:00.712103 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.712087    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:00.717122 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.717094    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78ee8-predictor-serving-cert\""
Jun 15 06:11:00.717438 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.717416    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-78ee8-kube-rbac-proxy-sar-config\""
Jun 15 06:11:00.731303 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.731218    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"]
Jun 15 06:11:00.849575 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.849535    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09814fd3-7783-461d-bcfd-b985deffc25b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:00.849743 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.849583    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft4jk\" (UniqueName: \"kubernetes.io/projected/09814fd3-7783-461d-bcfd-b985deffc25b-kube-api-access-ft4jk\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:00.849743 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.849671    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:00.950350 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.950316    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09814fd3-7783-461d-bcfd-b985deffc25b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:00.950350 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.950351    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft4jk\" (UniqueName: \"kubernetes.io/projected/09814fd3-7783-461d-bcfd-b985deffc25b-kube-api-access-ft4jk\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:00.950601 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.950404    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:00.950601 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:00.950527    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-serving-cert: secret "success-200-isvc-78ee8-predictor-serving-cert" not found
Jun 15 06:11:00.950601 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:00.950596    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls podName:09814fd3-7783-461d-bcfd-b985deffc25b nodeName:}" failed. No retries permitted until 2026-06-15 06:11:01.450579175 +0000 UTC m=+1404.403446103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls") pod "success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" (UID: "09814fd3-7783-461d-bcfd-b985deffc25b") : secret "success-200-isvc-78ee8-predictor-serving-cert" not found
Jun 15 06:11:00.951126 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.951102    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09814fd3-7783-461d-bcfd-b985deffc25b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:00.961232 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:00.961197    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft4jk\" (UniqueName: \"kubernetes.io/projected/09814fd3-7783-461d-bcfd-b985deffc25b-kube-api-access-ft4jk\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:01.152112 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:01.152014    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-serving-cert: secret "success-200-isvc-ea4bd-predictor-serving-cert" not found
Jun 15 06:11:01.152112 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:01.152095    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls podName:1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82 nodeName:}" failed. No retries permitted until 2026-06-15 06:11:02.152077354 +0000 UTC m=+1405.104944277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls") pod "success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" (UID: "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82") : secret "success-200-isvc-ea4bd-predictor-serving-cert" not found
Jun 15 06:11:01.167627 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:01.167589    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container" containerID="cri-o://c63fbcda38131d926e57e94e815fd1f7385e46114a409ebb9b5a9c17333f78f1" gracePeriod=30
Jun 15 06:11:01.167627 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:01.167618    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kube-rbac-proxy" containerID="cri-o://40f12168b42aecb014ea4d4b1b08388d54c7551961c2f21936e61630af176ed2" gracePeriod=30
Jun 15 06:11:01.453752 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:01.453641    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:01.456120 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:01.456090    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls\") pod \"success-200-isvc-78ee8-predictor-79b588b5d-jhzfw\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") " pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:01.624600 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:01.624554    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:01.768183 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:01.768156    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"]
Jun 15 06:11:01.770438 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:11:01.770408    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09814fd3_7783_461d_bcfd_b985deffc25b.slice/crio-dd36f937ecefa646ecabe9a5056e932bc8597934a898d468f4fcc6a40f259785 WatchSource:0}: Error finding container dd36f937ecefa646ecabe9a5056e932bc8597934a898d468f4fcc6a40f259785: Status 404 returned error can't find the container with id dd36f937ecefa646ecabe9a5056e932bc8597934a898d468f4fcc6a40f259785
Jun 15 06:11:02.160570 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:02.160529    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-serving-cert: secret "success-200-isvc-ea4bd-predictor-serving-cert" not found
Jun 15 06:11:02.160772 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:02.160607    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls podName:1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82 nodeName:}" failed. No retries permitted until 2026-06-15 06:11:04.160590105 +0000 UTC m=+1407.113457028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls") pod "success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" (UID: "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82") : secret "success-200-isvc-ea4bd-predictor-serving-cert" not found
Jun 15 06:11:02.174731 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.174701    2562 generic.go:356] "Generic (PLEG): container finished" podID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerID="40f12168b42aecb014ea4d4b1b08388d54c7551961c2f21936e61630af176ed2" exitCode=2
Jun 15 06:11:02.174887 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.174772    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" event={"ID":"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82","Type":"ContainerDied","Data":"40f12168b42aecb014ea4d4b1b08388d54c7551961c2f21936e61630af176ed2"}
Jun 15 06:11:02.176440 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.176413    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" event={"ID":"09814fd3-7783-461d-bcfd-b985deffc25b","Type":"ContainerStarted","Data":"e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514"}
Jun 15 06:11:02.176587 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.176468    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" event={"ID":"09814fd3-7783-461d-bcfd-b985deffc25b","Type":"ContainerStarted","Data":"6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e"}
Jun 15 06:11:02.176587 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.176502    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:02.176587 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.176529    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" event={"ID":"09814fd3-7783-461d-bcfd-b985deffc25b","Type":"ContainerStarted","Data":"dd36f937ecefa646ecabe9a5056e932bc8597934a898d468f4fcc6a40f259785"}
Jun 15 06:11:02.176587 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.176556    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:02.178066 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.178032    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jun 15 06:11:02.201178 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.201133    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podStartSLOduration=2.201120358 podStartE2EDuration="2.201120358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:11:02.198595807 +0000 UTC m=+1405.151462747" watchObservedRunningTime="2026-06-15 06:11:02.201120358 +0000 UTC m=+1405.153987382"
Jun 15 06:11:02.989375 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:02.989341    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:11:03.182351 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:03.182319    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jun 15 06:11:03.716252 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:03.716212    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jun 15 06:11:03.722269 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:03.722236    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jun 15 06:11:04.179329 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:04.179302    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-serving-cert: secret "success-200-isvc-ea4bd-predictor-serving-cert" not found
Jun 15 06:11:04.179698 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:04.179394    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls podName:1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82 nodeName:}" failed. No retries permitted until 2026-06-15 06:11:08.179363489 +0000 UTC m=+1411.132230411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls") pod "success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" (UID: "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82") : secret "success-200-isvc-ea4bd-predictor-serving-cert" not found
Jun 15 06:11:04.189623 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.189597    2562 generic.go:356] "Generic (PLEG): container finished" podID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerID="c63fbcda38131d926e57e94e815fd1f7385e46114a409ebb9b5a9c17333f78f1" exitCode=0
Jun 15 06:11:04.189720 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.189661    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" event={"ID":"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82","Type":"ContainerDied","Data":"c63fbcda38131d926e57e94e815fd1f7385e46114a409ebb9b5a9c17333f78f1"}
Jun 15 06:11:04.319558 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.319532    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:11:04.481806 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.481778    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-rbac-proxy-sar-config\") pod \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") "
Jun 15 06:11:04.481975 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.481858    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls\") pod \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") "
Jun 15 06:11:04.481975 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.481941    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4jq5\" (UniqueName: \"kubernetes.io/projected/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-api-access-p4jq5\") pod \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\" (UID: \"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82\") "
Jun 15 06:11:04.482179 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.482153    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" (UID: "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:04.483938 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.483915    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" (UID: "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:04.484157 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.484137    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-api-access-p4jq5" (OuterVolumeSpecName: "kube-api-access-p4jq5") pod "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" (UID: "1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82"). InnerVolumeSpecName "kube-api-access-p4jq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:11:04.583211 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.583182    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4jq5\" (UniqueName: \"kubernetes.io/projected/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-api-access-p4jq5\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:11:04.583211 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.583208    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:11:04.583211 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:04.583218    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:11:05.195880 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:05.195852    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"
Jun 15 06:11:05.196281 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:05.195838    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75" event={"ID":"1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82","Type":"ContainerDied","Data":"eeaf437164db1c24dc303401672c8a15c386c45c2a204b46e5637ced93c2dc7d"}
Jun 15 06:11:05.196281 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:05.195995    2562 scope.go:117] "RemoveContainer" containerID="40f12168b42aecb014ea4d4b1b08388d54c7551961c2f21936e61630af176ed2"
Jun 15 06:11:05.205426 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:05.205412    2562 scope.go:117] "RemoveContainer" containerID="c63fbcda38131d926e57e94e815fd1f7385e46114a409ebb9b5a9c17333f78f1"
Jun 15 06:11:05.225086 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:05.225061    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"]
Jun 15 06:11:05.234047 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:05.234026    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-ea4bd-predictor-65c4fc694d-sbg75"]
Jun 15 06:11:05.490390 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:05.490357    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" path="/var/lib/kubelet/pods/1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82/volumes"
Jun 15 06:11:07.989217 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:07.989175    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:11:08.187092 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:08.187066    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:08.187505 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:08.187477    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jun 15 06:11:12.989293 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:12.989248    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:11:12.989691 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:12.989356    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:11:17.989220 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:17.989175    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:11:18.188110 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:18.188069    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jun 15 06:11:22.989338 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:22.989296    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:11:27.989155 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:27.989109    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:11:28.187920 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:28.187885    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jun 15 06:11:30.666344 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:30.666316    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:11:30.695980 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:30.695944    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc762f-ac90-4697-a69a-70447f4c4eeb-openshift-service-ca-bundle\") pod \"8adc762f-ac90-4697-a69a-70447f4c4eeb\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") "
Jun 15 06:11:30.696139 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:30.696006    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls\") pod \"8adc762f-ac90-4697-a69a-70447f4c4eeb\" (UID: \"8adc762f-ac90-4697-a69a-70447f4c4eeb\") "
Jun 15 06:11:30.696368 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:30.696339    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adc762f-ac90-4697-a69a-70447f4c4eeb-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "8adc762f-ac90-4697-a69a-70447f4c4eeb" (UID: "8adc762f-ac90-4697-a69a-70447f4c4eeb"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:30.698225 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:30.698199    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8adc762f-ac90-4697-a69a-70447f4c4eeb" (UID: "8adc762f-ac90-4697-a69a-70447f4c4eeb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:30.796867 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:30.796792    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8adc762f-ac90-4697-a69a-70447f4c4eeb-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:11:30.796867 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:30.796820    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8adc762f-ac90-4697-a69a-70447f4c4eeb-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:11:31.313273 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.313239    2562 generic.go:356] "Generic (PLEG): container finished" podID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerID="b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c" exitCode=0
Jun 15 06:11:31.313495 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.313293    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"
Jun 15 06:11:31.313495 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.313326    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" event={"ID":"8adc762f-ac90-4697-a69a-70447f4c4eeb","Type":"ContainerDied","Data":"b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c"}
Jun 15 06:11:31.313495 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.313426    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n" event={"ID":"8adc762f-ac90-4697-a69a-70447f4c4eeb","Type":"ContainerDied","Data":"86d1cffaaa3160dc401a4facc1389f9b6eb0e1c3d1db1035959c52cb3caa2fce"}
Jun 15 06:11:31.313495 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.313450    2562 scope.go:117] "RemoveContainer" containerID="b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c"
Jun 15 06:11:31.323817 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.323802    2562 scope.go:117] "RemoveContainer" containerID="b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c"
Jun 15 06:11:31.324100 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:11:31.324077    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c\": container with ID starting with b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c not found: ID does not exist" containerID="b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c"
Jun 15 06:11:31.324164 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.324111    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c"} err="failed to get container status \"b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c\": rpc error: code = NotFound desc = could not find container \"b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c\": container with ID starting with b2134c71b3dde765c1a26d374aa2a6b5dc94f68d984cbad31a29f68db5217e6c not found: ID does not exist"
Jun 15 06:11:31.337298 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.337268    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"]
Jun 15 06:11:31.341392 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.341359    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-ea4bd-76d9f94847-lx78n"]
Jun 15 06:11:31.489859 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:31.489809    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" path="/var/lib/kubelet/pods/8adc762f-ac90-4697-a69a-70447f4c4eeb/volumes"
Jun 15 06:11:38.187504 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:38.187465    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jun 15 06:11:48.188290 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:48.188259    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:11:56.224991 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.224952    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"]
Jun 15 06:11:56.225421 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.225200    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a" containerID="cri-o://7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c" gracePeriod=30
Jun 15 06:11:56.352426 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.352364    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"]
Jun 15 06:11:56.352662 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.352634    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container" containerID="cri-o://fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16" gracePeriod=30
Jun 15 06:11:56.352744 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.352698    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kube-rbac-proxy" containerID="cri-o://c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60" gracePeriod=30
Jun 15 06:11:56.373263 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373236    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"]
Jun 15 06:11:56.373633 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373619    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container"
Jun 15 06:11:56.373693 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373636    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container"
Jun 15 06:11:56.373693 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373654    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kube-rbac-proxy"
Jun 15 06:11:56.373693 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373659    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kube-rbac-proxy"
Jun 15 06:11:56.373693 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373670    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd"
Jun 15 06:11:56.373693 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373675    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd"
Jun 15 06:11:56.373878 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373732    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kube-rbac-proxy"
Jun 15 06:11:56.373878 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373741    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e314429-bd2e-44b9-9c6e-f4fe2fbc0c82" containerName="kserve-container"
Jun 15 06:11:56.373878 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.373749    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adc762f-ac90-4697-a69a-70447f4c4eeb" containerName="switch-graph-ea4bd"
Jun 15 06:11:56.378132 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.378112    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.380987 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.380966    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-42a73-kube-rbac-proxy-sar-config\""
Jun 15 06:11:56.381074 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.380985    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-42a73-predictor-serving-cert\""
Jun 15 06:11:56.387346 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.387323    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"]
Jun 15 06:11:56.512717 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.512613    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.512717 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.512675    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nvtd\" (UniqueName: \"kubernetes.io/projected/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-api-access-2nvtd\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.512939 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.512739    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6910f7e-ced0-4964-aa33-b9b82d3bf482-proxy-tls\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.613345 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.613300    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.613345 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.613352    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nvtd\" (UniqueName: \"kubernetes.io/projected/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-api-access-2nvtd\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.613653 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.613436    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6910f7e-ced0-4964-aa33-b9b82d3bf482-proxy-tls\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.614015 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.613992    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.616031 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.616008    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6910f7e-ced0-4964-aa33-b9b82d3bf482-proxy-tls\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.623093 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.623073    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nvtd\" (UniqueName: \"kubernetes.io/projected/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-api-access-2nvtd\") pod \"success-200-isvc-42a73-predictor-5854874678-fr4b4\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") " pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.689631 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.689590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:56.827567 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:56.827538    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"]
Jun 15 06:11:56.830145 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:11:56.830120    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6910f7e_ced0_4964_aa33_b9b82d3bf482.slice/crio-e51b7b2dd66bae5e1bf2cd236a2989a846e233b045933c7da70d9caf01f038c6 WatchSource:0}: Error finding container e51b7b2dd66bae5e1bf2cd236a2989a846e233b045933c7da70d9caf01f038c6: Status 404 returned error can't find the container with id e51b7b2dd66bae5e1bf2cd236a2989a846e233b045933c7da70d9caf01f038c6
Jun 15 06:11:57.431914 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.431879    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" event={"ID":"c6910f7e-ced0-4964-aa33-b9b82d3bf482","Type":"ContainerStarted","Data":"d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43"}
Jun 15 06:11:57.432415 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.431942    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" event={"ID":"c6910f7e-ced0-4964-aa33-b9b82d3bf482","Type":"ContainerStarted","Data":"ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5"}
Jun 15 06:11:57.432415 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.431979    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:57.432415 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.432111    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:11:57.432415 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.432169    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" event={"ID":"c6910f7e-ced0-4964-aa33-b9b82d3bf482","Type":"ContainerStarted","Data":"e51b7b2dd66bae5e1bf2cd236a2989a846e233b045933c7da70d9caf01f038c6"}
Jun 15 06:11:57.433419 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.433364    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jun 15 06:11:57.433726 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.433706    2562 generic.go:356] "Generic (PLEG): container finished" podID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerID="c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60" exitCode=2
Jun 15 06:11:57.433805 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.433747    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" event={"ID":"5993ef3c-639e-43b7-9d21-d54e0c592262","Type":"ContainerDied","Data":"c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60"}
Jun 15 06:11:57.449659 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:57.449619    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podStartSLOduration=1.449603497 podStartE2EDuration="1.449603497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:11:57.448809456 +0000 UTC m=+1460.401676402" watchObservedRunningTime="2026-06-15 06:11:57.449603497 +0000 UTC m=+1460.402470439"
Jun 15 06:11:58.438830 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:58.438791    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jun 15 06:11:58.964899 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:58.964849    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jun 15 06:11:58.970529 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:58.970495    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jun 15 06:11:59.242715 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.242676    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:11:59.612470 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.612448    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:11:59.739071 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.739042    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5993ef3c-639e-43b7-9d21-d54e0c592262-proxy-tls\") pod \"5993ef3c-639e-43b7-9d21-d54e0c592262\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") "
Jun 15 06:11:59.739213 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.739092    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlcs5\" (UniqueName: \"kubernetes.io/projected/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-api-access-qlcs5\") pod \"5993ef3c-639e-43b7-9d21-d54e0c592262\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") "
Jun 15 06:11:59.739213 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.739133    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-rbac-proxy-sar-config\") pod \"5993ef3c-639e-43b7-9d21-d54e0c592262\" (UID: \"5993ef3c-639e-43b7-9d21-d54e0c592262\") "
Jun 15 06:11:59.739515 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.739491    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5993ef3c-639e-43b7-9d21-d54e0c592262" (UID: "5993ef3c-639e-43b7-9d21-d54e0c592262"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:59.741277 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.741250    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5993ef3c-639e-43b7-9d21-d54e0c592262-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5993ef3c-639e-43b7-9d21-d54e0c592262" (UID: "5993ef3c-639e-43b7-9d21-d54e0c592262"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:59.741336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.741304    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-api-access-qlcs5" (OuterVolumeSpecName: "kube-api-access-qlcs5") pod "5993ef3c-639e-43b7-9d21-d54e0c592262" (UID: "5993ef3c-639e-43b7-9d21-d54e0c592262"). InnerVolumeSpecName "kube-api-access-qlcs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:11:59.840369 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.840290    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlcs5\" (UniqueName: \"kubernetes.io/projected/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-api-access-qlcs5\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:11:59.840369 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.840317    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5993ef3c-639e-43b7-9d21-d54e0c592262-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:11:59.840369 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:11:59.840328    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5993ef3c-639e-43b7-9d21-d54e0c592262-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:12:00.448241 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.448207    2562 generic.go:356] "Generic (PLEG): container finished" podID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerID="fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16" exitCode=0
Jun 15 06:12:00.448434 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.448250    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" event={"ID":"5993ef3c-639e-43b7-9d21-d54e0c592262","Type":"ContainerDied","Data":"fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16"}
Jun 15 06:12:00.448434 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.448256    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"
Jun 15 06:12:00.448434 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.448299    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h" event={"ID":"5993ef3c-639e-43b7-9d21-d54e0c592262","Type":"ContainerDied","Data":"29eacc795ed8f9c028d680ed70cc64dbd005b500fa03089a35c073b53241aede"}
Jun 15 06:12:00.448434 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.448324    2562 scope.go:117] "RemoveContainer" containerID="c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60"
Jun 15 06:12:00.457742 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.457724    2562 scope.go:117] "RemoveContainer" containerID="fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16"
Jun 15 06:12:00.465203 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.464983    2562 scope.go:117] "RemoveContainer" containerID="c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60"
Jun 15 06:12:00.465284 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:12:00.465235    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60\": container with ID starting with c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60 not found: ID does not exist" containerID="c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60"
Jun 15 06:12:00.465348 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.465283    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60"} err="failed to get container status \"c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60\": rpc error: code = NotFound desc = could not find container \"c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60\": container with ID starting with c89c749862a7489f6b145d1d9939fdd810af8ff31339afadfda4147e11637f60 not found: ID does not exist"
Jun 15 06:12:00.465348 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.465299    2562 scope.go:117] "RemoveContainer" containerID="fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16"
Jun 15 06:12:00.465576 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:12:00.465556    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16\": container with ID starting with fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16 not found: ID does not exist" containerID="fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16"
Jun 15 06:12:00.465651 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.465577    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16"} err="failed to get container status \"fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16\": rpc error: code = NotFound desc = could not find container \"fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16\": container with ID starting with fce70cfbd1ef4268997577d3cc4920a1820841f1cdd4b6a4c9a429b7a0031b16 not found: ID does not exist"
Jun 15 06:12:00.474064 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.474033    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"]
Jun 15 06:12:00.478351 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.478328    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0ac5a-predictor-6867d6c9b-49h5h"]
Jun 15 06:12:00.740192 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.740152    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"]
Jun 15 06:12:00.740606 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.740568    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container"
Jun 15 06:12:00.740606 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.740582    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container"
Jun 15 06:12:00.740606 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.740599    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kube-rbac-proxy"
Jun 15 06:12:00.740606 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.740607    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kube-rbac-proxy"
Jun 15 06:12:00.740743 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.740693    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kserve-container"
Jun 15 06:12:00.740743 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.740709    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" containerName="kube-rbac-proxy"
Jun 15 06:12:00.745114 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.745098    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:00.747891 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.747871    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-78ee8-kube-rbac-proxy-sar-config\""
Jun 15 06:12:00.747891 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.747878    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-78ee8-serving-cert\""
Jun 15 06:12:00.754464 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.754439    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"]
Jun 15 06:12:00.851390 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.851330    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3305d51f-389c-485b-bbde-94986e6daa53-proxy-tls\") pod \"ensemble-graph-78ee8-85bfbcfdb7-hzl6c\" (UID: \"3305d51f-389c-485b-bbde-94986e6daa53\") " pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:00.851554 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.851403    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3305d51f-389c-485b-bbde-94986e6daa53-openshift-service-ca-bundle\") pod \"ensemble-graph-78ee8-85bfbcfdb7-hzl6c\" (UID: \"3305d51f-389c-485b-bbde-94986e6daa53\") " pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:00.952690 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.952646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3305d51f-389c-485b-bbde-94986e6daa53-proxy-tls\") pod \"ensemble-graph-78ee8-85bfbcfdb7-hzl6c\" (UID: \"3305d51f-389c-485b-bbde-94986e6daa53\") " pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:00.952690 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.952696    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3305d51f-389c-485b-bbde-94986e6daa53-openshift-service-ca-bundle\") pod \"ensemble-graph-78ee8-85bfbcfdb7-hzl6c\" (UID: \"3305d51f-389c-485b-bbde-94986e6daa53\") " pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:00.953262 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.953239    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3305d51f-389c-485b-bbde-94986e6daa53-openshift-service-ca-bundle\") pod \"ensemble-graph-78ee8-85bfbcfdb7-hzl6c\" (UID: \"3305d51f-389c-485b-bbde-94986e6daa53\") " pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:00.955200 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:00.955177    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3305d51f-389c-485b-bbde-94986e6daa53-proxy-tls\") pod \"ensemble-graph-78ee8-85bfbcfdb7-hzl6c\" (UID: \"3305d51f-389c-485b-bbde-94986e6daa53\") " pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:01.057498 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:01.057404    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:01.193306 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:01.193278    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"]
Jun 15 06:12:01.195914 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:12:01.195887    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3305d51f_389c_485b_bbde_94986e6daa53.slice/crio-1e04b0743f8d948f17c261c77235fec75ef18da70f76da48599369c35e1378fe WatchSource:0}: Error finding container 1e04b0743f8d948f17c261c77235fec75ef18da70f76da48599369c35e1378fe: Status 404 returned error can't find the container with id 1e04b0743f8d948f17c261c77235fec75ef18da70f76da48599369c35e1378fe
Jun 15 06:12:01.455171 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:01.455078    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" event={"ID":"3305d51f-389c-485b-bbde-94986e6daa53","Type":"ContainerStarted","Data":"8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec"}
Jun 15 06:12:01.455171 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:01.455130    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" event={"ID":"3305d51f-389c-485b-bbde-94986e6daa53","Type":"ContainerStarted","Data":"1e04b0743f8d948f17c261c77235fec75ef18da70f76da48599369c35e1378fe"}
Jun 15 06:12:01.455376 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:01.455217    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:01.473233 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:01.473174    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" podStartSLOduration=1.473158977 podStartE2EDuration="1.473158977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:12:01.471958282 +0000 UTC m=+1464.424825226" watchObservedRunningTime="2026-06-15 06:12:01.473158977 +0000 UTC m=+1464.426025969"
Jun 15 06:12:01.489879 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:01.489847    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5993ef3c-639e-43b7-9d21-d54e0c592262" path="/var/lib/kubelet/pods/5993ef3c-639e-43b7-9d21-d54e0c592262/volumes"
Jun 15 06:12:03.444031 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:03.443999    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:12:03.444504 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:03.444401    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jun 15 06:12:04.243315 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:04.243274    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:07.470902 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:07.470873    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:09.244226 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:09.244172    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:09.244817 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:09.244308    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:12:10.806069 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.806033    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"]
Jun 15 06:12:10.806474 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.806241    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8" containerID="cri-o://8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec" gracePeriod=30
Jun 15 06:12:10.928026 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.927982    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"]
Jun 15 06:12:10.928247 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.928216    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container" containerID="cri-o://6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e" gracePeriod=30
Jun 15 06:12:10.928415 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.928256    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kube-rbac-proxy" containerID="cri-o://e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514" gracePeriod=30
Jun 15 06:12:10.942892 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.942861    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"]
Jun 15 06:12:10.946073 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.946054    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:10.949631 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.949601    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b50b1-predictor-serving-cert\""
Jun 15 06:12:10.950327 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.950309    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-b50b1-kube-rbac-proxy-sar-config\""
Jun 15 06:12:10.961101 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:10.961071    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"]
Jun 15 06:12:11.045634 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.045604    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-proxy-tls\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.045771 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.045651    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7gz\" (UniqueName: \"kubernetes.io/projected/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-api-access-rb7gz\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.045813 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.045769    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.146937 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.146852    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.146937 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.146914    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-proxy-tls\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.147168 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.146953    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7gz\" (UniqueName: \"kubernetes.io/projected/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-api-access-rb7gz\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.147697 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.147615    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.149462 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.149440    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-proxy-tls\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.157321 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.157298    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7gz\" (UniqueName: \"kubernetes.io/projected/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-api-access-rb7gz\") pod \"success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") " pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.256306 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.256268    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:11.413829 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.413804    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"]
Jun 15 06:12:11.416331 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:12:11.416303    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf6bf62_6c07_4b14_bbcd_f43cb0c4d046.slice/crio-f734f772b7edc5caf22929ba7e2662b10bd2a7a9f09be521b4b200685d28dd11 WatchSource:0}: Error finding container f734f772b7edc5caf22929ba7e2662b10bd2a7a9f09be521b4b200685d28dd11: Status 404 returned error can't find the container with id f734f772b7edc5caf22929ba7e2662b10bd2a7a9f09be521b4b200685d28dd11
Jun 15 06:12:11.508011 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.507982    2562 generic.go:356] "Generic (PLEG): container finished" podID="09814fd3-7783-461d-bcfd-b985deffc25b" containerID="e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514" exitCode=2
Jun 15 06:12:11.508197 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.508056    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" event={"ID":"09814fd3-7783-461d-bcfd-b985deffc25b","Type":"ContainerDied","Data":"e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514"}
Jun 15 06:12:11.509810 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.509666    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" event={"ID":"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046","Type":"ContainerStarted","Data":"d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b"}
Jun 15 06:12:11.509810 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:11.509725    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" event={"ID":"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046","Type":"ContainerStarted","Data":"f734f772b7edc5caf22929ba7e2662b10bd2a7a9f09be521b4b200685d28dd11"}
Jun 15 06:12:12.470477 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:12.470434    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:12.515782 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:12.515746    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" event={"ID":"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046","Type":"ContainerStarted","Data":"eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615"}
Jun 15 06:12:12.515972 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:12.515856    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:12.535373 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:12.535331    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podStartSLOduration=2.535317188 podStartE2EDuration="2.535317188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:12:12.534270694 +0000 UTC m=+1475.487137640" watchObservedRunningTime="2026-06-15 06:12:12.535317188 +0000 UTC m=+1475.488184132"
Jun 15 06:12:13.183038 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:13.182991    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jun 15 06:12:13.445053 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:13.444960    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jun 15 06:12:13.520755 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:13.520725    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:13.522093 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:13.522065    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jun 15 06:12:14.243732 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.243683    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:14.286691 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.286669    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:12:14.369677 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.369582    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ft4jk\" (UniqueName: \"kubernetes.io/projected/09814fd3-7783-461d-bcfd-b985deffc25b-kube-api-access-ft4jk\") pod \"09814fd3-7783-461d-bcfd-b985deffc25b\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") "
Jun 15 06:12:14.369677 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.369644    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls\") pod \"09814fd3-7783-461d-bcfd-b985deffc25b\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") "
Jun 15 06:12:14.371891 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.371849    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09814fd3-7783-461d-bcfd-b985deffc25b-kube-api-access-ft4jk" (OuterVolumeSpecName: "kube-api-access-ft4jk") pod "09814fd3-7783-461d-bcfd-b985deffc25b" (UID: "09814fd3-7783-461d-bcfd-b985deffc25b"). InnerVolumeSpecName "kube-api-access-ft4jk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:12:14.372005 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.371856    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09814fd3-7783-461d-bcfd-b985deffc25b" (UID: "09814fd3-7783-461d-bcfd-b985deffc25b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:12:14.470705 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.470672    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09814fd3-7783-461d-bcfd-b985deffc25b-kube-rbac-proxy-sar-config\") pod \"09814fd3-7783-461d-bcfd-b985deffc25b\" (UID: \"09814fd3-7783-461d-bcfd-b985deffc25b\") "
Jun 15 06:12:14.470896 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.470880    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft4jk\" (UniqueName: \"kubernetes.io/projected/09814fd3-7783-461d-bcfd-b985deffc25b-kube-api-access-ft4jk\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:12:14.470945 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.470904    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09814fd3-7783-461d-bcfd-b985deffc25b-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:12:14.471132 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.471101    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09814fd3-7783-461d-bcfd-b985deffc25b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09814fd3-7783-461d-bcfd-b985deffc25b" (UID: "09814fd3-7783-461d-bcfd-b985deffc25b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:12:14.526583 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.526544    2562 generic.go:356] "Generic (PLEG): container finished" podID="09814fd3-7783-461d-bcfd-b985deffc25b" containerID="6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e" exitCode=0
Jun 15 06:12:14.527030 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.526593    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"
Jun 15 06:12:14.527030 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.526626    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" event={"ID":"09814fd3-7783-461d-bcfd-b985deffc25b","Type":"ContainerDied","Data":"6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e"}
Jun 15 06:12:14.527030 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.526685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw" event={"ID":"09814fd3-7783-461d-bcfd-b985deffc25b","Type":"ContainerDied","Data":"dd36f937ecefa646ecabe9a5056e932bc8597934a898d468f4fcc6a40f259785"}
Jun 15 06:12:14.527030 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.526712    2562 scope.go:117] "RemoveContainer" containerID="e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514"
Jun 15 06:12:14.527237 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.527217    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jun 15 06:12:14.535555 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.535536    2562 scope.go:117] "RemoveContainer" containerID="6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e"
Jun 15 06:12:14.547876 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.547844    2562 scope.go:117] "RemoveContainer" containerID="e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514"
Jun 15 06:12:14.548244 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:12:14.548217    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514\": container with ID starting with e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514 not found: ID does not exist" containerID="e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514"
Jun 15 06:12:14.548298 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.548265    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514"} err="failed to get container status \"e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514\": rpc error: code = NotFound desc = could not find container \"e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514\": container with ID starting with e328cbcc9e2c4a32479b017e91e952a02244e886191282f4bf6ac1a834df7514 not found: ID does not exist"
Jun 15 06:12:14.548298 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.548288    2562 scope.go:117] "RemoveContainer" containerID="6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e"
Jun 15 06:12:14.548661 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:12:14.548631    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e\": container with ID starting with 6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e not found: ID does not exist" containerID="6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e"
Jun 15 06:12:14.548753 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.548686    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e"} err="failed to get container status \"6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e\": rpc error: code = NotFound desc = could not find container \"6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e\": container with ID starting with 6fbd407251af3eceb2d2300b68f2c1b0cc1e0fd01b208e8a1f12e3c8495ff67e not found: ID does not exist"
Jun 15 06:12:14.551015 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.550995    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"]
Jun 15 06:12:14.554631 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.554608    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-78ee8-predictor-79b588b5d-jhzfw"]
Jun 15 06:12:14.571667 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:14.571637    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09814fd3-7783-461d-bcfd-b985deffc25b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:12:15.490945 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:15.490912    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" path="/var/lib/kubelet/pods/09814fd3-7783-461d-bcfd-b985deffc25b/volumes"
Jun 15 06:12:17.469249 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:17.469204    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:19.243437 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:19.243368    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:19.531579 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:19.531497    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:12:19.532003 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:19.531977    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jun 15 06:12:22.469651 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:22.469598    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:22.470044 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:22.469746    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:23.444956 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:23.444903    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jun 15 06:12:24.242909 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:24.242866    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:26.384909 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.384887    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:12:26.448743 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.448712    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f98403b-2ccc-48a2-a20c-806943d57b04-openshift-service-ca-bundle\") pod \"6f98403b-2ccc-48a2-a20c-806943d57b04\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") "
Jun 15 06:12:26.448880 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.448754    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls\") pod \"6f98403b-2ccc-48a2-a20c-806943d57b04\" (UID: \"6f98403b-2ccc-48a2-a20c-806943d57b04\") "
Jun 15 06:12:26.449070 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.449047    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f98403b-2ccc-48a2-a20c-806943d57b04-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "6f98403b-2ccc-48a2-a20c-806943d57b04" (UID: "6f98403b-2ccc-48a2-a20c-806943d57b04"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:12:26.450982 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.450958    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f98403b-2ccc-48a2-a20c-806943d57b04" (UID: "6f98403b-2ccc-48a2-a20c-806943d57b04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:12:26.549709 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.549611    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f98403b-2ccc-48a2-a20c-806943d57b04-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:12:26.549709 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.549646    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f98403b-2ccc-48a2-a20c-806943d57b04-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:12:26.584424 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.584375    2562 generic.go:356] "Generic (PLEG): container finished" podID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerID="7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c" exitCode=0
Jun 15 06:12:26.584573 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.584436    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"
Jun 15 06:12:26.584573 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.584466    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" event={"ID":"6f98403b-2ccc-48a2-a20c-806943d57b04","Type":"ContainerDied","Data":"7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c"}
Jun 15 06:12:26.584573 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.584508    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp" event={"ID":"6f98403b-2ccc-48a2-a20c-806943d57b04","Type":"ContainerDied","Data":"e188cb7d9a82c147301d863f77f178d18698e98f6152143ecdba79a9dd22ac8d"}
Jun 15 06:12:26.584573 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.584529    2562 scope.go:117] "RemoveContainer" containerID="7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c"
Jun 15 06:12:26.593310 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.593294    2562 scope.go:117] "RemoveContainer" containerID="7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c"
Jun 15 06:12:26.593582 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:12:26.593565    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c\": container with ID starting with 7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c not found: ID does not exist" containerID="7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c"
Jun 15 06:12:26.593627 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.593592    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c"} err="failed to get container status \"7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c\": rpc error: code = NotFound desc = could not find container \"7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c\": container with ID starting with 7fffae28f56b9f3e6190627a07bdc17ebaf9ce6eafe576ea5fe02553ffcd361c not found: ID does not exist"
Jun 15 06:12:26.605669 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.605645    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"]
Jun 15 06:12:26.609035 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:26.609008    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-0ac5a-665fcb9dd4-hxvgp"]
Jun 15 06:12:27.470067 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:27.470026    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:27.489395 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:27.489360    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" path="/var/lib/kubelet/pods/6f98403b-2ccc-48a2-a20c-806943d57b04/volumes"
Jun 15 06:12:29.532190 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:29.532149    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jun 15 06:12:32.469008 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:32.468970    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:33.445336 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:33.445294    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jun 15 06:12:37.469364 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:37.469317    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:12:37.543096 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:37.543066    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:12:37.545746 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:37.545726    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:12:39.532950 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:39.532903    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jun 15 06:12:40.958918 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:40.958891    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:41.055154 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.055122    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3305d51f-389c-485b-bbde-94986e6daa53-proxy-tls\") pod \"3305d51f-389c-485b-bbde-94986e6daa53\" (UID: \"3305d51f-389c-485b-bbde-94986e6daa53\") "
Jun 15 06:12:41.055341 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.055179    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3305d51f-389c-485b-bbde-94986e6daa53-openshift-service-ca-bundle\") pod \"3305d51f-389c-485b-bbde-94986e6daa53\" (UID: \"3305d51f-389c-485b-bbde-94986e6daa53\") "
Jun 15 06:12:41.055599 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.055568    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3305d51f-389c-485b-bbde-94986e6daa53-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3305d51f-389c-485b-bbde-94986e6daa53" (UID: "3305d51f-389c-485b-bbde-94986e6daa53"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:12:41.057194 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.057173    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3305d51f-389c-485b-bbde-94986e6daa53-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3305d51f-389c-485b-bbde-94986e6daa53" (UID: "3305d51f-389c-485b-bbde-94986e6daa53"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:12:41.156060 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.156023    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3305d51f-389c-485b-bbde-94986e6daa53-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:12:41.156060 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.156055    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3305d51f-389c-485b-bbde-94986e6daa53-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:12:41.652176 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.652138    2562 generic.go:356] "Generic (PLEG): container finished" podID="3305d51f-389c-485b-bbde-94986e6daa53" containerID="8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec" exitCode=0
Jun 15 06:12:41.652176 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.652179    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"
Jun 15 06:12:41.652423 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.652225    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" event={"ID":"3305d51f-389c-485b-bbde-94986e6daa53","Type":"ContainerDied","Data":"8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec"}
Jun 15 06:12:41.652423 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.652271    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c" event={"ID":"3305d51f-389c-485b-bbde-94986e6daa53","Type":"ContainerDied","Data":"1e04b0743f8d948f17c261c77235fec75ef18da70f76da48599369c35e1378fe"}
Jun 15 06:12:41.652423 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.652294    2562 scope.go:117] "RemoveContainer" containerID="8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec"
Jun 15 06:12:41.660429 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.660414    2562 scope.go:117] "RemoveContainer" containerID="8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec"
Jun 15 06:12:41.660688 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:12:41.660670    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec\": container with ID starting with 8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec not found: ID does not exist" containerID="8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec"
Jun 15 06:12:41.660744 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.660696    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec"} err="failed to get container status \"8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec\": rpc error: code = NotFound desc = could not find container \"8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec\": container with ID starting with 8a3fc99bf60cd58374c3496ec6b5b8c99f91fef95b430616fd62e14fdfb479ec not found: ID does not exist"
Jun 15 06:12:41.669265 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.669237    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"]
Jun 15 06:12:41.673019 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:41.672992    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-78ee8-85bfbcfdb7-hzl6c"]
Jun 15 06:12:43.445403 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:43.445364    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:12:43.492407 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:43.492357    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3305d51f-389c-485b-bbde-94986e6daa53" path="/var/lib/kubelet/pods/3305d51f-389c-485b-bbde-94986e6daa53/volumes"
Jun 15 06:12:49.532685 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:49.532638    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jun 15 06:12:56.403055 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.402972    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"]
Jun 15 06:12:56.403417 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403297    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a"
Jun 15 06:12:56.403417 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403309    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a"
Jun 15 06:12:56.403417 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403323    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kube-rbac-proxy"
Jun 15 06:12:56.403417 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403329    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kube-rbac-proxy"
Jun 15 06:12:56.403417 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403338    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container"
Jun 15 06:12:56.403417 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403343    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container"
Jun 15 06:12:56.403417 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403355    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8"
Jun 15 06:12:56.403417 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403361    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8"
Jun 15 06:12:56.403708 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403434    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f98403b-2ccc-48a2-a20c-806943d57b04" containerName="sequence-graph-0ac5a"
Jun 15 06:12:56.403708 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403444    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kube-rbac-proxy"
Jun 15 06:12:56.403708 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403449    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3305d51f-389c-485b-bbde-94986e6daa53" containerName="ensemble-graph-78ee8"
Jun 15 06:12:56.403708 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.403455    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="09814fd3-7783-461d-bcfd-b985deffc25b" containerName="kserve-container"
Jun 15 06:12:56.407753 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.407732    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:56.410442 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.410416    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-42a73-kube-rbac-proxy-sar-config\""
Jun 15 06:12:56.410571 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.410438    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-42a73-serving-cert\""
Jun 15 06:12:56.416537 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.416509    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"]
Jun 15 06:12:56.482156 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.482119    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a6942c9-6b46-45b5-ab87-54bdb5428385-proxy-tls\") pod \"sequence-graph-42a73-84c6b98785-vsxjq\" (UID: \"4a6942c9-6b46-45b5-ab87-54bdb5428385\") " pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:56.482353 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.482180    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a6942c9-6b46-45b5-ab87-54bdb5428385-openshift-service-ca-bundle\") pod \"sequence-graph-42a73-84c6b98785-vsxjq\" (UID: \"4a6942c9-6b46-45b5-ab87-54bdb5428385\") " pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:56.582928 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.582885    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a6942c9-6b46-45b5-ab87-54bdb5428385-proxy-tls\") pod \"sequence-graph-42a73-84c6b98785-vsxjq\" (UID: \"4a6942c9-6b46-45b5-ab87-54bdb5428385\") " pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:56.583136 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.582979    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a6942c9-6b46-45b5-ab87-54bdb5428385-openshift-service-ca-bundle\") pod \"sequence-graph-42a73-84c6b98785-vsxjq\" (UID: \"4a6942c9-6b46-45b5-ab87-54bdb5428385\") " pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:56.583713 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.583692    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a6942c9-6b46-45b5-ab87-54bdb5428385-openshift-service-ca-bundle\") pod \"sequence-graph-42a73-84c6b98785-vsxjq\" (UID: \"4a6942c9-6b46-45b5-ab87-54bdb5428385\") " pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:56.585559 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.585534    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a6942c9-6b46-45b5-ab87-54bdb5428385-proxy-tls\") pod \"sequence-graph-42a73-84c6b98785-vsxjq\" (UID: \"4a6942c9-6b46-45b5-ab87-54bdb5428385\") " pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:56.719446 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.719334    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:56.858213 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:56.858168    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"]
Jun 15 06:12:57.725230 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:57.725193    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" event={"ID":"4a6942c9-6b46-45b5-ab87-54bdb5428385","Type":"ContainerStarted","Data":"7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480"}
Jun 15 06:12:57.725647 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:57.725253    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" event={"ID":"4a6942c9-6b46-45b5-ab87-54bdb5428385","Type":"ContainerStarted","Data":"798959d158c101b187131a187ae09230721e5dc7b525a5b2c6950841edac1b9c"}
Jun 15 06:12:57.725647 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:57.725291    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:12:57.742898 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:57.742849    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" podStartSLOduration=1.742832748 podStartE2EDuration="1.742832748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:12:57.741858325 +0000 UTC m=+1520.694725271" watchObservedRunningTime="2026-06-15 06:12:57.742832748 +0000 UTC m=+1520.695699693"
Jun 15 06:12:59.532529 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:12:59.532500    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:13:03.735450 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:03.735419    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:13:06.514226 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.514195    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"]
Jun 15 06:13:06.514657 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.514408    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73" containerID="cri-o://7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480" gracePeriod=30
Jun 15 06:13:06.632274 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.632230    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"]
Jun 15 06:13:06.634327 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.634275    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container" containerID="cri-o://ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5" gracePeriod=30
Jun 15 06:13:06.637545 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.636964    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"]
Jun 15 06:13:06.638722 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.637185    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kube-rbac-proxy" containerID="cri-o://d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43" gracePeriod=30
Jun 15 06:13:06.643999 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.643980    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:06.647787 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.647763    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-44470-predictor-serving-cert\""
Jun 15 06:13:06.648317 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.648297    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-44470-kube-rbac-proxy-sar-config\""
Jun 15 06:13:06.651542 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.651518    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"]
Jun 15 06:13:06.770202 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.770110    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:06.770202 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.770178    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:06.770455 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.770276    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b66wd\" (UniqueName: \"kubernetes.io/projected/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-api-access-b66wd\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:06.871486 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.871450    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:06.871705 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.871515    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:06.871705 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.871545    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b66wd\" (UniqueName: \"kubernetes.io/projected/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-api-access-b66wd\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:06.871705 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:13:06.871669    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-44470-predictor-serving-cert: secret "success-200-isvc-44470-predictor-serving-cert" not found
Jun 15 06:13:06.871877 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:13:06.871756    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls podName:33cd3e46-5e4b-457a-b235-ea70b284a6b0 nodeName:}" failed. No retries permitted until 2026-06-15 06:13:07.371735865 +0000 UTC m=+1530.324602788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls") pod "success-200-isvc-44470-predictor-5787bff68b-pcp4k" (UID: "33cd3e46-5e4b-457a-b235-ea70b284a6b0") : secret "success-200-isvc-44470-predictor-serving-cert" not found
Jun 15 06:13:06.872262 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.872236    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:06.885595 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:06.885565    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b66wd\" (UniqueName: \"kubernetes.io/projected/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-api-access-b66wd\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:07.376475 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:07.376441    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:07.378967 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:07.378944    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls\") pod \"success-200-isvc-44470-predictor-5787bff68b-pcp4k\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") " pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:07.571636 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:07.571590    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:07.710076 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:07.709987    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"]
Jun 15 06:13:07.712964 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:13:07.712936    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33cd3e46_5e4b_457a_b235_ea70b284a6b0.slice/crio-8c9ff1594ed25d69df4a0ba0d6a06ec36ed267ef82692d154779ab6e80bb6e6a WatchSource:0}: Error finding container 8c9ff1594ed25d69df4a0ba0d6a06ec36ed267ef82692d154779ab6e80bb6e6a: Status 404 returned error can't find the container with id 8c9ff1594ed25d69df4a0ba0d6a06ec36ed267ef82692d154779ab6e80bb6e6a
Jun 15 06:13:07.778193 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:07.778166    2562 generic.go:356] "Generic (PLEG): container finished" podID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerID="d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43" exitCode=2
Jun 15 06:13:07.778290 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:07.778228    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" event={"ID":"c6910f7e-ced0-4964-aa33-b9b82d3bf482","Type":"ContainerDied","Data":"d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43"}
Jun 15 06:13:07.779816 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:07.779795    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" event={"ID":"33cd3e46-5e4b-457a-b235-ea70b284a6b0","Type":"ContainerStarted","Data":"8c9ff1594ed25d69df4a0ba0d6a06ec36ed267ef82692d154779ab6e80bb6e6a"}
Jun 15 06:13:08.439442 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:08.439364    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jun 15 06:13:08.734172 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:08.734136    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:13:08.786613 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:08.786576    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" event={"ID":"33cd3e46-5e4b-457a-b235-ea70b284a6b0","Type":"ContainerStarted","Data":"9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a"}
Jun 15 06:13:08.786821 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:08.786641    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" event={"ID":"33cd3e46-5e4b-457a-b235-ea70b284a6b0","Type":"ContainerStarted","Data":"795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7"}
Jun 15 06:13:08.786821 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:08.786684    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:08.786821 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:08.786803    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:08.788106 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:08.788083    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jun 15 06:13:08.807722 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:08.807669    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podStartSLOduration=2.807654362 podStartE2EDuration="2.807654362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:13:08.804908473 +0000 UTC m=+1531.757775443" watchObservedRunningTime="2026-06-15 06:13:08.807654362 +0000 UTC m=+1531.760521307"
Jun 15 06:13:09.791527 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:09.791482    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jun 15 06:13:09.994213 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:09.994186    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:13:10.102370 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.102272    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6910f7e-ced0-4964-aa33-b9b82d3bf482-proxy-tls\") pod \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") "
Jun 15 06:13:10.102370 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.102333    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2nvtd\" (UniqueName: \"kubernetes.io/projected/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-api-access-2nvtd\") pod \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") "
Jun 15 06:13:10.102638 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.102425    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-rbac-proxy-sar-config\") pod \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\" (UID: \"c6910f7e-ced0-4964-aa33-b9b82d3bf482\") "
Jun 15 06:13:10.102824 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.102795    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6910f7e-ced0-4964-aa33-b9b82d3bf482" (UID: "c6910f7e-ced0-4964-aa33-b9b82d3bf482"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:13:10.104618 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.104587    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6910f7e-ced0-4964-aa33-b9b82d3bf482-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6910f7e-ced0-4964-aa33-b9b82d3bf482" (UID: "c6910f7e-ced0-4964-aa33-b9b82d3bf482"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:13:10.104691 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.104619    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-api-access-2nvtd" (OuterVolumeSpecName: "kube-api-access-2nvtd") pod "c6910f7e-ced0-4964-aa33-b9b82d3bf482" (UID: "c6910f7e-ced0-4964-aa33-b9b82d3bf482"). InnerVolumeSpecName "kube-api-access-2nvtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:13:10.203206 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.203173    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6910f7e-ced0-4964-aa33-b9b82d3bf482-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:13:10.203206 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.203201    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nvtd\" (UniqueName: \"kubernetes.io/projected/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-api-access-2nvtd\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:13:10.203206 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.203210    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6910f7e-ced0-4964-aa33-b9b82d3bf482-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:13:10.797400 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.797342    2562 generic.go:356] "Generic (PLEG): container finished" podID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerID="ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5" exitCode=0
Jun 15 06:13:10.797834 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.797417    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" event={"ID":"c6910f7e-ced0-4964-aa33-b9b82d3bf482","Type":"ContainerDied","Data":"ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5"}
Jun 15 06:13:10.797834 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.797480    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4" event={"ID":"c6910f7e-ced0-4964-aa33-b9b82d3bf482","Type":"ContainerDied","Data":"e51b7b2dd66bae5e1bf2cd236a2989a846e233b045933c7da70d9caf01f038c6"}
Jun 15 06:13:10.797834 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.797419    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"
Jun 15 06:13:10.797834 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.797507    2562 scope.go:117] "RemoveContainer" containerID="d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43"
Jun 15 06:13:10.806448 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.806424    2562 scope.go:117] "RemoveContainer" containerID="ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5"
Jun 15 06:13:10.814012 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.813992    2562 scope.go:117] "RemoveContainer" containerID="d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43"
Jun 15 06:13:10.814272 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:13:10.814255    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43\": container with ID starting with d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43 not found: ID does not exist" containerID="d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43"
Jun 15 06:13:10.814312 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.814281    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43"} err="failed to get container status \"d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43\": rpc error: code = NotFound desc = could not find container \"d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43\": container with ID starting with d577508e16bc1393554945ae74403ddd499b28c405ed97af1e89844a256e6f43 not found: ID does not exist"
Jun 15 06:13:10.814312 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.814298    2562 scope.go:117] "RemoveContainer" containerID="ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5"
Jun 15 06:13:10.814543 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:13:10.814526    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5\": container with ID starting with ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5 not found: ID does not exist" containerID="ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5"
Jun 15 06:13:10.814586 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.814547    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5"} err="failed to get container status \"ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5\": rpc error: code = NotFound desc = could not find container \"ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5\": container with ID starting with ba5333617aa2de492f64fdfcaf883a03e52f6cb4cfef25cbf1a562502b0c91b5 not found: ID does not exist"
Jun 15 06:13:10.820115 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.820090    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"]
Jun 15 06:13:10.822317 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:10.822294    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-42a73-predictor-5854874678-fr4b4"]
Jun 15 06:13:11.004285 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.004245    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"]
Jun 15 06:13:11.004852 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.004834    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container"
Jun 15 06:13:11.004908 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.004856    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container"
Jun 15 06:13:11.004908 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.004874    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kube-rbac-proxy"
Jun 15 06:13:11.004908 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.004882    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kube-rbac-proxy"
Jun 15 06:13:11.005012 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.004950    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kube-rbac-proxy"
Jun 15 06:13:11.005012 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.004962    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" containerName="kserve-container"
Jun 15 06:13:11.009525 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.009500    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:11.012118 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.012091    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b50b1-kube-rbac-proxy-sar-config\""
Jun 15 06:13:11.012256 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.012098    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-b50b1-serving-cert\""
Jun 15 06:13:11.017660 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.017635    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"]
Jun 15 06:13:11.110820 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.110736    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2e122f-76c2-49e6-8401-8af1f72d500f-openshift-service-ca-bundle\") pod \"ensemble-graph-b50b1-97587847f-nkhzb\" (UID: \"2c2e122f-76c2-49e6-8401-8af1f72d500f\") " pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:11.110820 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.110772    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e122f-76c2-49e6-8401-8af1f72d500f-proxy-tls\") pod \"ensemble-graph-b50b1-97587847f-nkhzb\" (UID: \"2c2e122f-76c2-49e6-8401-8af1f72d500f\") " pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:11.211200 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.211164    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2e122f-76c2-49e6-8401-8af1f72d500f-openshift-service-ca-bundle\") pod \"ensemble-graph-b50b1-97587847f-nkhzb\" (UID: \"2c2e122f-76c2-49e6-8401-8af1f72d500f\") " pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:11.211200 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.211201    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e122f-76c2-49e6-8401-8af1f72d500f-proxy-tls\") pod \"ensemble-graph-b50b1-97587847f-nkhzb\" (UID: \"2c2e122f-76c2-49e6-8401-8af1f72d500f\") " pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:11.211857 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.211827    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2e122f-76c2-49e6-8401-8af1f72d500f-openshift-service-ca-bundle\") pod \"ensemble-graph-b50b1-97587847f-nkhzb\" (UID: \"2c2e122f-76c2-49e6-8401-8af1f72d500f\") " pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:11.213744 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.213723    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e122f-76c2-49e6-8401-8af1f72d500f-proxy-tls\") pod \"ensemble-graph-b50b1-97587847f-nkhzb\" (UID: \"2c2e122f-76c2-49e6-8401-8af1f72d500f\") " pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:11.322065 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.322024    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:11.459159 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.459133    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"]
Jun 15 06:13:11.461588 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:13:11.461563    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2e122f_76c2_49e6_8401_8af1f72d500f.slice/crio-d6e48885236e45fcdc4b8e16f2177a574cf6c7d3e322394bd6fe57b597c1bf1f WatchSource:0}: Error finding container d6e48885236e45fcdc4b8e16f2177a574cf6c7d3e322394bd6fe57b597c1bf1f: Status 404 returned error can't find the container with id d6e48885236e45fcdc4b8e16f2177a574cf6c7d3e322394bd6fe57b597c1bf1f
Jun 15 06:13:11.489142 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.489114    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6910f7e-ced0-4964-aa33-b9b82d3bf482" path="/var/lib/kubelet/pods/c6910f7e-ced0-4964-aa33-b9b82d3bf482/volumes"
Jun 15 06:13:11.803682 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.803647    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" event={"ID":"2c2e122f-76c2-49e6-8401-8af1f72d500f","Type":"ContainerStarted","Data":"246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001"}
Jun 15 06:13:11.804086 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.803698    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" event={"ID":"2c2e122f-76c2-49e6-8401-8af1f72d500f","Type":"ContainerStarted","Data":"d6e48885236e45fcdc4b8e16f2177a574cf6c7d3e322394bd6fe57b597c1bf1f"}
Jun 15 06:13:11.804086 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:11.803770    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:13.733765 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:13.733727    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:13:14.796479 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:14.796445    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:13:14.796877 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:14.796851    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jun 15 06:13:14.816860 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:14.816812    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" podStartSLOduration=4.81679804 podStartE2EDuration="4.81679804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:13:11.826925715 +0000 UTC m=+1534.779792660" watchObservedRunningTime="2026-06-15 06:13:14.81679804 +0000 UTC m=+1537.769664988"
Jun 15 06:13:17.815124 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:17.815094    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:13:18.735276 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:18.735231    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:13:18.735476 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:18.735348    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:13:23.734352 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:23.734313    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:13:24.797561 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:24.797522    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jun 15 06:13:28.734274 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:28.734233    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:13:33.733870 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:33.733833    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:13:34.797325 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:34.797282    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jun 15 06:13:36.672248 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.672222    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:13:36.726682 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.726647    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a6942c9-6b46-45b5-ab87-54bdb5428385-openshift-service-ca-bundle\") pod \"4a6942c9-6b46-45b5-ab87-54bdb5428385\" (UID: \"4a6942c9-6b46-45b5-ab87-54bdb5428385\") "
Jun 15 06:13:36.726889 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.726698    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a6942c9-6b46-45b5-ab87-54bdb5428385-proxy-tls\") pod \"4a6942c9-6b46-45b5-ab87-54bdb5428385\" (UID: \"4a6942c9-6b46-45b5-ab87-54bdb5428385\") "
Jun 15 06:13:36.727082 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.727054    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6942c9-6b46-45b5-ab87-54bdb5428385-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4a6942c9-6b46-45b5-ab87-54bdb5428385" (UID: "4a6942c9-6b46-45b5-ab87-54bdb5428385"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:13:36.728935 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.728910    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6942c9-6b46-45b5-ab87-54bdb5428385-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a6942c9-6b46-45b5-ab87-54bdb5428385" (UID: "4a6942c9-6b46-45b5-ab87-54bdb5428385"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:13:36.827810 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.827714    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a6942c9-6b46-45b5-ab87-54bdb5428385-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:13:36.827810 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.827745    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a6942c9-6b46-45b5-ab87-54bdb5428385-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:13:36.916067 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.916038    2562 generic.go:356] "Generic (PLEG): container finished" podID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerID="7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480" exitCode=0
Jun 15 06:13:36.916238 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.916088    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"
Jun 15 06:13:36.916238 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.916129    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" event={"ID":"4a6942c9-6b46-45b5-ab87-54bdb5428385","Type":"ContainerDied","Data":"7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480"}
Jun 15 06:13:36.916238 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.916215    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq" event={"ID":"4a6942c9-6b46-45b5-ab87-54bdb5428385","Type":"ContainerDied","Data":"798959d158c101b187131a187ae09230721e5dc7b525a5b2c6950841edac1b9c"}
Jun 15 06:13:36.916338 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.916242    2562 scope.go:117] "RemoveContainer" containerID="7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480"
Jun 15 06:13:36.925616 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.925596    2562 scope.go:117] "RemoveContainer" containerID="7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480"
Jun 15 06:13:36.925932 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:13:36.925910    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480\": container with ID starting with 7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480 not found: ID does not exist" containerID="7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480"
Jun 15 06:13:36.925990 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.925941    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480"} err="failed to get container status \"7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480\": rpc error: code = NotFound desc = could not find container \"7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480\": container with ID starting with 7fa57187602f59c379c82e930656281fde320b8c1e2c18758365fd0184296480 not found: ID does not exist"
Jun 15 06:13:36.939125 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.939090    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"]
Jun 15 06:13:36.945494 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:36.945464    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-42a73-84c6b98785-vsxjq"]
Jun 15 06:13:37.491635 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:37.491605    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" path="/var/lib/kubelet/pods/4a6942c9-6b46-45b5-ab87-54bdb5428385/volumes"
Jun 15 06:13:44.797512 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:44.797465    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jun 15 06:13:54.797569 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:13:54.797535    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:14:06.730349 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.730305    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"]
Jun 15 06:14:06.730869 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.730847    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73"
Jun 15 06:14:06.730869 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.730867    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73"
Jun 15 06:14:06.730984 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.730945    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a6942c9-6b46-45b5-ab87-54bdb5428385" containerName="sequence-graph-42a73"
Jun 15 06:14:06.733723 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.733703    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:06.736432 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.736401    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-44470-serving-cert\""
Jun 15 06:14:06.736723 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.736700    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-44470-kube-rbac-proxy-sar-config\""
Jun 15 06:14:06.743499 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.743470    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"]
Jun 15 06:14:06.781513 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.781470    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-openshift-service-ca-bundle\") pod \"sequence-graph-44470-76d55687fd-l8kcg\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") " pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:06.781711 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.781595    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls\") pod \"sequence-graph-44470-76d55687fd-l8kcg\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") " pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:06.882118 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.882074    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-openshift-service-ca-bundle\") pod \"sequence-graph-44470-76d55687fd-l8kcg\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") " pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:06.882295 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.882141    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls\") pod \"sequence-graph-44470-76d55687fd-l8kcg\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") " pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:06.882295 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:14:06.882260    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-44470-serving-cert: secret "sequence-graph-44470-serving-cert" not found
Jun 15 06:14:06.882374 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:14:06.882346    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls podName:fe4fe338-5d3d-4d5d-a911-91ef4d98afc2 nodeName:}" failed. No retries permitted until 2026-06-15 06:14:07.382330787 +0000 UTC m=+1590.335197709 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls") pod "sequence-graph-44470-76d55687fd-l8kcg" (UID: "fe4fe338-5d3d-4d5d-a911-91ef4d98afc2") : secret "sequence-graph-44470-serving-cert" not found
Jun 15 06:14:06.882769 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:06.882746    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-openshift-service-ca-bundle\") pod \"sequence-graph-44470-76d55687fd-l8kcg\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") " pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:07.386862 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:07.386822    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls\") pod \"sequence-graph-44470-76d55687fd-l8kcg\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") " pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:07.389418 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:07.389366    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls\") pod \"sequence-graph-44470-76d55687fd-l8kcg\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") " pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:07.644963 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:07.644882    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:07.796980 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:07.796953    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"]
Jun 15 06:14:07.799686 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:14:07.799661    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4fe338_5d3d_4d5d_a911_91ef4d98afc2.slice/crio-2e5b8e3e95995aba397feae3132849f6dc865c099856308f6a50373068c1e742 WatchSource:0}: Error finding container 2e5b8e3e95995aba397feae3132849f6dc865c099856308f6a50373068c1e742: Status 404 returned error can't find the container with id 2e5b8e3e95995aba397feae3132849f6dc865c099856308f6a50373068c1e742
Jun 15 06:14:08.054373 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:08.054338    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" event={"ID":"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2","Type":"ContainerStarted","Data":"7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e"}
Jun 15 06:14:08.054574 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:08.054415    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" event={"ID":"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2","Type":"ContainerStarted","Data":"2e5b8e3e95995aba397feae3132849f6dc865c099856308f6a50373068c1e742"}
Jun 15 06:14:08.054574 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:08.054495    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:14:08.073790 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:08.073742    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" podStartSLOduration=2.073727214 podStartE2EDuration="2.073727214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:14:08.073305813 +0000 UTC m=+1591.026172757" watchObservedRunningTime="2026-06-15 06:14:08.073727214 +0000 UTC m=+1591.026594158"
Jun 15 06:14:14.064259 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:14:14.064228    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:17:37.568832 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:17:37.568800    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:17:37.572766 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:17:37.572746    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:21:25.708920 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.708885    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"]
Jun 15 06:21:25.711371 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.709140    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1" containerID="cri-o://246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001" gracePeriod=30
Jun 15 06:21:25.807808 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.807771    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"]
Jun 15 06:21:25.808103 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.808073    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container" containerID="cri-o://d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b" gracePeriod=30
Jun 15 06:21:25.808258 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.808116    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kube-rbac-proxy" containerID="cri-o://eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615" gracePeriod=30
Jun 15 06:21:25.894057 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.894022    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"]
Jun 15 06:21:25.897614 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.897594    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:25.901175 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.900603    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5c5b6-predictor-serving-cert\""
Jun 15 06:21:25.901175 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.900867    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5c5b6-kube-rbac-proxy-sar-config\""
Jun 15 06:21:25.909222 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.909167    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"]
Jun 15 06:21:25.979177 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.979139    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:25.979345 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.979185    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:25.979345 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:25.979256    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn9jm\" (UniqueName: \"kubernetes.io/projected/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-api-access-hn9jm\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.006842 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.006807    2562 generic.go:356] "Generic (PLEG): container finished" podID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerID="eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615" exitCode=2
Jun 15 06:21:26.007030 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.006879    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" event={"ID":"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046","Type":"ContainerDied","Data":"eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615"}
Jun 15 06:21:26.080322 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.080281    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.080558 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.080331    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.080558 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.080414    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9jm\" (UniqueName: \"kubernetes.io/projected/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-api-access-hn9jm\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.080558 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:21:26.080476    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-serving-cert: secret "success-200-isvc-5c5b6-predictor-serving-cert" not found
Jun 15 06:21:26.080558 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:21:26.080547    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls podName:b87b98ed-848e-44ec-b2a8-f2b5bf5a222d nodeName:}" failed. No retries permitted until 2026-06-15 06:21:26.58052748 +0000 UTC m=+2029.533394409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls") pod "success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" (UID: "b87b98ed-848e-44ec-b2a8-f2b5bf5a222d") : secret "success-200-isvc-5c5b6-predictor-serving-cert" not found
Jun 15 06:21:26.081065 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.081047    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.089303 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.089268    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9jm\" (UniqueName: \"kubernetes.io/projected/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-api-access-hn9jm\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.584598 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.584549    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.587216 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.587192    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls\") pod \"success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") " pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.811306 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.811265    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:26.955286 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:26.955260    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"]
Jun 15 06:21:26.957026 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:21:26.956996    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb87b98ed_848e_44ec_b2a8_f2b5bf5a222d.slice/crio-0e86075a2556d7bf13837f3ee521c454a33d45c422aea79424539aeba963ddaa WatchSource:0}: Error finding container 0e86075a2556d7bf13837f3ee521c454a33d45c422aea79424539aeba963ddaa: Status 404 returned error can't find the container with id 0e86075a2556d7bf13837f3ee521c454a33d45c422aea79424539aeba963ddaa
Jun 15 06:21:27.013549 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:27.013522    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" event={"ID":"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d","Type":"ContainerStarted","Data":"0e86075a2556d7bf13837f3ee521c454a33d45c422aea79424539aeba963ddaa"}
Jun 15 06:21:27.813178 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:27.813138    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:21:28.019089 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:28.019050    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" event={"ID":"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d","Type":"ContainerStarted","Data":"770b3e983fb2fa5d19f4df7bc1a71e4ee918c3b44fd810ba15081756e4ce9477"}
Jun 15 06:21:28.019274 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:28.019110    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" event={"ID":"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d","Type":"ContainerStarted","Data":"500b191afafa133f032753c6fe55816b9d56e73b78a130ae00f65866f185da55"}
Jun 15 06:21:28.019274 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:28.019165    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:28.051801 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:28.051744    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podStartSLOduration=3.051728517 podStartE2EDuration="3.051728517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:21:28.051650864 +0000 UTC m=+2031.004517810" watchObservedRunningTime="2026-06-15 06:21:28.051728517 +0000 UTC m=+2031.004595461"
Jun 15 06:21:29.023222 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.023187    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:29.024618 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.024588    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jun 15 06:21:29.167308 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.167282    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:21:29.204784 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.204747    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-proxy-tls\") pod \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") "
Jun 15 06:21:29.204941 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.204803    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb7gz\" (UniqueName: \"kubernetes.io/projected/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-api-access-rb7gz\") pod \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") "
Jun 15 06:21:29.204941 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.204827    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-rbac-proxy-sar-config\") pod \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\" (UID: \"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046\") "
Jun 15 06:21:29.205256 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.205232    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" (UID: "1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:21:29.207020 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.206997    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" (UID: "1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:21:29.207020 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.207002    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-api-access-rb7gz" (OuterVolumeSpecName: "kube-api-access-rb7gz") pod "1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" (UID: "1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046"). InnerVolumeSpecName "kube-api-access-rb7gz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:21:29.305408 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.305369    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:21:29.305408 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.305406    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7gz\" (UniqueName: \"kubernetes.io/projected/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-api-access-rb7gz\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:21:29.305408 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:29.305415    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:21:30.029071 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.029034    2562 generic.go:356] "Generic (PLEG): container finished" podID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerID="d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b" exitCode=0
Jun 15 06:21:30.029540 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.029080    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"
Jun 15 06:21:30.029540 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.029119    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" event={"ID":"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046","Type":"ContainerDied","Data":"d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b"}
Jun 15 06:21:30.029540 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.029176    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n" event={"ID":"1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046","Type":"ContainerDied","Data":"f734f772b7edc5caf22929ba7e2662b10bd2a7a9f09be521b4b200685d28dd11"}
Jun 15 06:21:30.029540 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.029200    2562 scope.go:117] "RemoveContainer" containerID="eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615"
Jun 15 06:21:30.029868 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.029840    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jun 15 06:21:30.037682 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.037658    2562 scope.go:117] "RemoveContainer" containerID="d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b"
Jun 15 06:21:30.045329 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.045305    2562 scope.go:117] "RemoveContainer" containerID="eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615"
Jun 15 06:21:30.045667 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:21:30.045624    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615\": container with ID starting with eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615 not found: ID does not exist" containerID="eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615"
Jun 15 06:21:30.045795 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.045666    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615"} err="failed to get container status \"eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615\": rpc error: code = NotFound desc = could not find container \"eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615\": container with ID starting with eed6458040c4fb7bad01136a741d7be87ce39dcb9575c013c0d10e0d83dfe615 not found: ID does not exist"
Jun 15 06:21:30.045795 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.045690    2562 scope.go:117] "RemoveContainer" containerID="d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b"
Jun 15 06:21:30.046076 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:21:30.046050    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b\": container with ID starting with d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b not found: ID does not exist" containerID="d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b"
Jun 15 06:21:30.046137 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.046084    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b"} err="failed to get container status \"d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b\": rpc error: code = NotFound desc = could not find container \"d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b\": container with ID starting with d572f424b05c2c0afd75dbe5fcf4d2c897f9991a92a83962f891ab5688f8099b not found: ID does not exist"
Jun 15 06:21:30.047516 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.047497    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"]
Jun 15 06:21:30.051710 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:30.051687    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-b50b1-predictor-d45d95b7b-nkv5n"]
Jun 15 06:21:31.489936 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:31.489905    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" path="/var/lib/kubelet/pods/1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046/volumes"
Jun 15 06:21:32.813256 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:32.813210    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:21:35.034349 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:35.034322    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:21:35.034807 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:35.034779    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jun 15 06:21:37.813778 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:37.813737    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:21:37.814150 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:37.813848    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:21:42.813400 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:42.813341    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:21:45.035134 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:45.035091    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jun 15 06:21:47.813732 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:47.813694    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:21:52.813657 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:52.813614    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:21:55.035473 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:55.035365    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jun 15 06:21:55.854366 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:55.854340    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:21:55.907242 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:55.907201    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2e122f-76c2-49e6-8401-8af1f72d500f-openshift-service-ca-bundle\") pod \"2c2e122f-76c2-49e6-8401-8af1f72d500f\" (UID: \"2c2e122f-76c2-49e6-8401-8af1f72d500f\") "
Jun 15 06:21:55.907440 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:55.907288    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e122f-76c2-49e6-8401-8af1f72d500f-proxy-tls\") pod \"2c2e122f-76c2-49e6-8401-8af1f72d500f\" (UID: \"2c2e122f-76c2-49e6-8401-8af1f72d500f\") "
Jun 15 06:21:55.907608 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:55.907587    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2e122f-76c2-49e6-8401-8af1f72d500f-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2c2e122f-76c2-49e6-8401-8af1f72d500f" (UID: "2c2e122f-76c2-49e6-8401-8af1f72d500f"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:21:55.909612 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:55.909590    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2e122f-76c2-49e6-8401-8af1f72d500f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c2e122f-76c2-49e6-8401-8af1f72d500f" (UID: "2c2e122f-76c2-49e6-8401-8af1f72d500f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:21:56.008112 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.008069    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c2e122f-76c2-49e6-8401-8af1f72d500f-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:21:56.008112 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.008102    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c2e122f-76c2-49e6-8401-8af1f72d500f-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:21:56.150903 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.150863    2562 generic.go:356] "Generic (PLEG): container finished" podID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerID="246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001" exitCode=0
Jun 15 06:21:56.150903 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.150906    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"
Jun 15 06:21:56.151329 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.150905    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" event={"ID":"2c2e122f-76c2-49e6-8401-8af1f72d500f","Type":"ContainerDied","Data":"246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001"}
Jun 15 06:21:56.151329 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.151027    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb" event={"ID":"2c2e122f-76c2-49e6-8401-8af1f72d500f","Type":"ContainerDied","Data":"d6e48885236e45fcdc4b8e16f2177a574cf6c7d3e322394bd6fe57b597c1bf1f"}
Jun 15 06:21:56.151329 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.151056    2562 scope.go:117] "RemoveContainer" containerID="246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001"
Jun 15 06:21:56.160184 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.160162    2562 scope.go:117] "RemoveContainer" containerID="246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001"
Jun 15 06:21:56.160477 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:21:56.160460    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001\": container with ID starting with 246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001 not found: ID does not exist" containerID="246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001"
Jun 15 06:21:56.160536 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.160485    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001"} err="failed to get container status \"246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001\": rpc error: code = NotFound desc = could not find container \"246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001\": container with ID starting with 246668c53bf5d0dacdb90c492ea5e7567a5bd6e1490bdb6c0fad11efaeb5c001 not found: ID does not exist"
Jun 15 06:21:56.172485 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.172457    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"]
Jun 15 06:21:56.175832 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:56.175808    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-b50b1-97587847f-nkhzb"]
Jun 15 06:21:57.489299 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:21:57.489256    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" path="/var/lib/kubelet/pods/2c2e122f-76c2-49e6-8401-8af1f72d500f/volumes"
Jun 15 06:22:05.035054 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:05.035013    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jun 15 06:22:15.039468 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:15.039426    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:22:21.507282 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.507246    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"]
Jun 15 06:22:21.507784 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.507484    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470" containerID="cri-o://7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e" gracePeriod=30
Jun 15 06:22:21.626357 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.626320    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"]
Jun 15 06:22:21.626606 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.626578    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container" containerID="cri-o://9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a" gracePeriod=30
Jun 15 06:22:21.627098 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.626620    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kube-rbac-proxy" containerID="cri-o://795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7" gracePeriod=30
Jun 15 06:22:21.661156 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661119    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"]
Jun 15 06:22:21.661652 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661634    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kube-rbac-proxy"
Jun 15 06:22:21.661652 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661653    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kube-rbac-proxy"
Jun 15 06:22:21.661796 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661663    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1"
Jun 15 06:22:21.661796 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661669    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1"
Jun 15 06:22:21.661796 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661684    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container"
Jun 15 06:22:21.661796 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661692    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container"
Jun 15 06:22:21.661796 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661767    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kserve-container"
Jun 15 06:22:21.661796 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661790    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2e122f-76c2-49e6-8401-8af1f72d500f" containerName="ensemble-graph-b50b1"
Jun 15 06:22:21.662103 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.661799    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cf6bf62-6c07-4b14-bbcd-f43cb0c4d046" containerName="kube-rbac-proxy"
Jun 15 06:22:21.664844 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.664824    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:21.667362 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.667343    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6c800-kube-rbac-proxy-sar-config\""
Jun 15 06:22:21.667481 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.667345    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-6c800-predictor-serving-cert\""
Jun 15 06:22:21.675286 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.675259    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"]
Jun 15 06:22:21.727517 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.727466    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2pwp\" (UniqueName: \"kubernetes.io/projected/71fd1f5e-b994-4010-90dd-b1158035917a-kube-api-access-x2pwp\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:21.727705 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.727535    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71fd1f5e-b994-4010-90dd-b1158035917a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:21.727705 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.727643    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:21.828130 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.828023    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:21.828130 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.828112    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2pwp\" (UniqueName: \"kubernetes.io/projected/71fd1f5e-b994-4010-90dd-b1158035917a-kube-api-access-x2pwp\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:21.828367 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.828155    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71fd1f5e-b994-4010-90dd-b1158035917a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:21.828367 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:21.828168    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-6c800-predictor-serving-cert: secret "success-200-isvc-6c800-predictor-serving-cert" not found
Jun 15 06:22:21.828367 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:21.828233    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls podName:71fd1f5e-b994-4010-90dd-b1158035917a nodeName:}" failed. No retries permitted until 2026-06-15 06:22:22.328212423 +0000 UTC m=+2085.281079348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls") pod "success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" (UID: "71fd1f5e-b994-4010-90dd-b1158035917a") : secret "success-200-isvc-6c800-predictor-serving-cert" not found
Jun 15 06:22:21.828880 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.828860    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71fd1f5e-b994-4010-90dd-b1158035917a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:21.836838 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:21.836811    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2pwp\" (UniqueName: \"kubernetes.io/projected/71fd1f5e-b994-4010-90dd-b1158035917a-kube-api-access-x2pwp\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:22.270542 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:22.270507    2562 generic.go:356] "Generic (PLEG): container finished" podID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerID="795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7" exitCode=2
Jun 15 06:22:22.270714 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:22.270579    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" event={"ID":"33cd3e46-5e4b-457a-b235-ea70b284a6b0","Type":"ContainerDied","Data":"795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7"}
Jun 15 06:22:22.332614 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:22.332580    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:22.335262 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:22.335237    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls\") pod \"success-200-isvc-6c800-predictor-866b9bb7cd-p55wh\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") " pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:22.577113 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:22.577017    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:22.726439 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:22.726374    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"]
Jun 15 06:22:22.729707 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:22:22.729677    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fd1f5e_b994_4010_90dd_b1158035917a.slice/crio-52b1f6a29b58da23537e03227f8bcb194b453f8e54daf85e281163c94d27900a WatchSource:0}: Error finding container 52b1f6a29b58da23537e03227f8bcb194b453f8e54daf85e281163c94d27900a: Status 404 returned error can't find the container with id 52b1f6a29b58da23537e03227f8bcb194b453f8e54daf85e281163c94d27900a
Jun 15 06:22:23.278663 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:23.278628    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" event={"ID":"71fd1f5e-b994-4010-90dd-b1158035917a","Type":"ContainerStarted","Data":"fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300"}
Jun 15 06:22:23.278852 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:23.278685    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" event={"ID":"71fd1f5e-b994-4010-90dd-b1158035917a","Type":"ContainerStarted","Data":"10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888"}
Jun 15 06:22:23.278852 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:23.278713    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" event={"ID":"71fd1f5e-b994-4010-90dd-b1158035917a","Type":"ContainerStarted","Data":"52b1f6a29b58da23537e03227f8bcb194b453f8e54daf85e281163c94d27900a"}
Jun 15 06:22:23.278852 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:23.278785    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:23.297867 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:23.297813    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podStartSLOduration=2.297798726 podStartE2EDuration="2.297798726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:22:23.296338305 +0000 UTC m=+2086.249205250" watchObservedRunningTime="2026-06-15 06:22:23.297798726 +0000 UTC m=+2086.250665670"
Jun 15 06:22:24.063264 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:24.063213    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:24.282889 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:24.282860    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:24.284037 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:24.284006    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jun 15 06:22:24.792008 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:24.791972    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jun 15 06:22:24.797413 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:24.797374    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jun 15 06:22:24.978652 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:24.978625    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:22:25.058349 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.058258    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-rbac-proxy-sar-config\") pod \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") "
Jun 15 06:22:25.058349 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.058315    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls\") pod \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") "
Jun 15 06:22:25.058553 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.058365    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b66wd\" (UniqueName: \"kubernetes.io/projected/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-api-access-b66wd\") pod \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\" (UID: \"33cd3e46-5e4b-457a-b235-ea70b284a6b0\") "
Jun 15 06:22:25.058653 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.058631    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "33cd3e46-5e4b-457a-b235-ea70b284a6b0" (UID: "33cd3e46-5e4b-457a-b235-ea70b284a6b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:22:25.060638 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.060603    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "33cd3e46-5e4b-457a-b235-ea70b284a6b0" (UID: "33cd3e46-5e4b-457a-b235-ea70b284a6b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:22:25.060757 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.060661    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-api-access-b66wd" (OuterVolumeSpecName: "kube-api-access-b66wd") pod "33cd3e46-5e4b-457a-b235-ea70b284a6b0" (UID: "33cd3e46-5e4b-457a-b235-ea70b284a6b0"). InnerVolumeSpecName "kube-api-access-b66wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:22:25.158940 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.158891    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/33cd3e46-5e4b-457a-b235-ea70b284a6b0-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:22:25.158940 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.158930    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b66wd\" (UniqueName: \"kubernetes.io/projected/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-api-access-b66wd\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:22:25.158940 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.158941    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/33cd3e46-5e4b-457a-b235-ea70b284a6b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:22:25.288287 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.288252    2562 generic.go:356] "Generic (PLEG): container finished" podID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerID="9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a" exitCode=0
Jun 15 06:22:25.288486 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.288301    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"
Jun 15 06:22:25.288486 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.288345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" event={"ID":"33cd3e46-5e4b-457a-b235-ea70b284a6b0","Type":"ContainerDied","Data":"9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a"}
Jun 15 06:22:25.288486 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.288443    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k" event={"ID":"33cd3e46-5e4b-457a-b235-ea70b284a6b0","Type":"ContainerDied","Data":"8c9ff1594ed25d69df4a0ba0d6a06ec36ed267ef82692d154779ab6e80bb6e6a"}
Jun 15 06:22:25.288486 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.288479    2562 scope.go:117] "RemoveContainer" containerID="795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7"
Jun 15 06:22:25.288987 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.288959    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jun 15 06:22:25.297535 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.297519    2562 scope.go:117] "RemoveContainer" containerID="9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a"
Jun 15 06:22:25.304977 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.304958    2562 scope.go:117] "RemoveContainer" containerID="795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7"
Jun 15 06:22:25.305218 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:25.305199    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7\": container with ID starting with 795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7 not found: ID does not exist" containerID="795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7"
Jun 15 06:22:25.305270 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.305226    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7"} err="failed to get container status \"795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7\": rpc error: code = NotFound desc = could not find container \"795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7\": container with ID starting with 795c5b399ebec55660b804edf754db1f04ab5c2208f2d257f558b05c93cb34d7 not found: ID does not exist"
Jun 15 06:22:25.305270 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.305242    2562 scope.go:117] "RemoveContainer" containerID="9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a"
Jun 15 06:22:25.305490 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:25.305473    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a\": container with ID starting with 9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a not found: ID does not exist" containerID="9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a"
Jun 15 06:22:25.305553 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.305493    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a"} err="failed to get container status \"9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a\": rpc error: code = NotFound desc = could not find container \"9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a\": container with ID starting with 9520c19a5e1485bc980f3aec6536a296dbb04a721723fa163a3023b44af03a3a not found: ID does not exist"
Jun 15 06:22:25.315977 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.315919    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"]
Jun 15 06:22:25.327554 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.327511    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-44470-predictor-5787bff68b-pcp4k"]
Jun 15 06:22:25.489432 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.489372    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" path="/var/lib/kubelet/pods/33cd3e46-5e4b-457a-b235-ea70b284a6b0/volumes"
Jun 15 06:22:25.931209 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.931178    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"]
Jun 15 06:22:25.931576 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.931562    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container"
Jun 15 06:22:25.931622 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.931578    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container"
Jun 15 06:22:25.931622 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.931592    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kube-rbac-proxy"
Jun 15 06:22:25.931622 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.931598    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kube-rbac-proxy"
Jun 15 06:22:25.931721 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.931675    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kserve-container"
Jun 15 06:22:25.931721 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.931685    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="33cd3e46-5e4b-457a-b235-ea70b284a6b0" containerName="kube-rbac-proxy"
Jun 15 06:22:25.936535 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.936509    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:25.939000 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.938975    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5c5b6-kube-rbac-proxy-sar-config\""
Jun 15 06:22:25.939161 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.938998    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5c5b6-serving-cert\""
Jun 15 06:22:25.943221 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:25.943185    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"]
Jun 15 06:22:26.067516 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.067475    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-openshift-service-ca-bundle\") pod \"splitter-graph-5c5b6-745649d7db-gtzw9\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") " pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:26.067516 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.067521    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls\") pod \"splitter-graph-5c5b6-745649d7db-gtzw9\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") " pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:26.168721 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.168671    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-openshift-service-ca-bundle\") pod \"splitter-graph-5c5b6-745649d7db-gtzw9\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") " pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:26.169196 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.168739    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls\") pod \"splitter-graph-5c5b6-745649d7db-gtzw9\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") " pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:26.169196 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:26.168886    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-5c5b6-serving-cert: secret "splitter-graph-5c5b6-serving-cert" not found
Jun 15 06:22:26.169196 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:26.168951    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls podName:bd74db9f-d5d3-4f88-aaf8-6b6417c07424 nodeName:}" failed. No retries permitted until 2026-06-15 06:22:26.668934863 +0000 UTC m=+2089.621801786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls") pod "splitter-graph-5c5b6-745649d7db-gtzw9" (UID: "bd74db9f-d5d3-4f88-aaf8-6b6417c07424") : secret "splitter-graph-5c5b6-serving-cert" not found
Jun 15 06:22:26.169373 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.169260    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-openshift-service-ca-bundle\") pod \"splitter-graph-5c5b6-745649d7db-gtzw9\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") " pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:26.673987 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.673949    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls\") pod \"splitter-graph-5c5b6-745649d7db-gtzw9\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") " pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:26.676448 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.676430    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls\") pod \"splitter-graph-5c5b6-745649d7db-gtzw9\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") " pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:26.849353 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.849314    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:26.987668 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:26.987642    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"]
Jun 15 06:22:26.990219 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:22:26.990186    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd74db9f_d5d3_4f88_aaf8_6b6417c07424.slice/crio-bff5f7028e8a6d8178f806f193031950317e8bf71f9f908b8b1b7b2d935b4710 WatchSource:0}: Error finding container bff5f7028e8a6d8178f806f193031950317e8bf71f9f908b8b1b7b2d935b4710: Status 404 returned error can't find the container with id bff5f7028e8a6d8178f806f193031950317e8bf71f9f908b8b1b7b2d935b4710
Jun 15 06:22:27.300720 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:27.300681    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" event={"ID":"bd74db9f-d5d3-4f88-aaf8-6b6417c07424","Type":"ContainerStarted","Data":"4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68"}
Jun 15 06:22:27.301149 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:27.300742    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" event={"ID":"bd74db9f-d5d3-4f88-aaf8-6b6417c07424","Type":"ContainerStarted","Data":"bff5f7028e8a6d8178f806f193031950317e8bf71f9f908b8b1b7b2d935b4710"}
Jun 15 06:22:27.301149 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:27.300816    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:27.318583 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:27.318530    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" podStartSLOduration=2.318512357 podStartE2EDuration="2.318512357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:22:27.316672721 +0000 UTC m=+2090.269539667" watchObservedRunningTime="2026-06-15 06:22:27.318512357 +0000 UTC m=+2090.271379301"
Jun 15 06:22:29.062919 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:29.062880    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:30.294081 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:30.294052    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:22:30.294617 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:30.294589    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jun 15 06:22:33.310606 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:33.310573    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:34.062198 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:34.062157    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:34.062430 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:34.062253    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:22:36.035027 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.034994    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"]
Jun 15 06:22:36.035481 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.035186    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6" containerID="cri-o://4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68" gracePeriod=30
Jun 15 06:22:36.127271 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.127239    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"]
Jun 15 06:22:36.127522 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.127495    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container" containerID="cri-o://500b191afafa133f032753c6fe55816b9d56e73b78a130ae00f65866f185da55" gracePeriod=30
Jun 15 06:22:36.127606 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.127531    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kube-rbac-proxy" containerID="cri-o://770b3e983fb2fa5d19f4df7bc1a71e4ee918c3b44fd810ba15081756e4ce9477" gracePeriod=30
Jun 15 06:22:36.152150 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.152118    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"]
Jun 15 06:22:36.155936 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.155914    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.158563 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.158542    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5edeb-kube-rbac-proxy-sar-config\""
Jun 15 06:22:36.158676 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.158543    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5edeb-predictor-serving-cert\""
Jun 15 06:22:36.165081 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.165055    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"]
Jun 15 06:22:36.255806 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.255772    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkwfq\" (UniqueName: \"kubernetes.io/projected/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-api-access-pkwfq\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.255961 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.255831    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.255961 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.255895    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.341919 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.341823    2562 generic.go:356] "Generic (PLEG): container finished" podID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerID="770b3e983fb2fa5d19f4df7bc1a71e4ee918c3b44fd810ba15081756e4ce9477" exitCode=2
Jun 15 06:22:36.341919 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.341890    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" event={"ID":"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d","Type":"ContainerDied","Data":"770b3e983fb2fa5d19f4df7bc1a71e4ee918c3b44fd810ba15081756e4ce9477"}
Jun 15 06:22:36.356276 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.356246    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkwfq\" (UniqueName: \"kubernetes.io/projected/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-api-access-pkwfq\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.356466 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.356319    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.356466 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.356354    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.356588 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:36.356532    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-serving-cert: secret "success-200-isvc-5edeb-predictor-serving-cert" not found
Jun 15 06:22:36.356640 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:36.356605    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls podName:d0fec994-cf65-43fc-b0a1-e5db0b83cc94 nodeName:}" failed. No retries permitted until 2026-06-15 06:22:36.856585554 +0000 UTC m=+2099.809452490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls") pod "success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" (UID: "d0fec994-cf65-43fc-b0a1-e5db0b83cc94") : secret "success-200-isvc-5edeb-predictor-serving-cert" not found
Jun 15 06:22:36.357070 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.357052    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.365278 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.365252    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkwfq\" (UniqueName: \"kubernetes.io/projected/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-api-access-pkwfq\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.860696 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.860655    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:36.863279 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:36.863248    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls\") pod \"success-200-isvc-5edeb-predictor-869ffb65f4-m57w9\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") " pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:37.068404 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.068339    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:37.207961 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.207933    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"]
Jun 15 06:22:37.210663 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:22:37.210633    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0fec994_cf65_43fc_b0a1_e5db0b83cc94.slice/crio-46a9c05e457c0ae41099fb442795ddb1ca529a196ed9b5142db4a11eb9a923e5 WatchSource:0}: Error finding container 46a9c05e457c0ae41099fb442795ddb1ca529a196ed9b5142db4a11eb9a923e5: Status 404 returned error can't find the container with id 46a9c05e457c0ae41099fb442795ddb1ca529a196ed9b5142db4a11eb9a923e5
Jun 15 06:22:37.348681 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.348652    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" event={"ID":"d0fec994-cf65-43fc-b0a1-e5db0b83cc94","Type":"ContainerStarted","Data":"64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db"}
Jun 15 06:22:37.348825 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.348710    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" event={"ID":"d0fec994-cf65-43fc-b0a1-e5db0b83cc94","Type":"ContainerStarted","Data":"6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc"}
Jun 15 06:22:37.348825 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.348730    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" event={"ID":"d0fec994-cf65-43fc-b0a1-e5db0b83cc94","Type":"ContainerStarted","Data":"46a9c05e457c0ae41099fb442795ddb1ca529a196ed9b5142db4a11eb9a923e5"}
Jun 15 06:22:37.348825 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.348771    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:37.366718 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.366658    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podStartSLOduration=1.36663774 podStartE2EDuration="1.36663774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:22:37.365197361 +0000 UTC m=+2100.318064334" watchObservedRunningTime="2026-06-15 06:22:37.36663774 +0000 UTC m=+2100.319504688"
Jun 15 06:22:37.597069 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.597044    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:22:37.602849 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:37.602826    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:22:38.309261 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:38.309222    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:38.353078 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:38.353046    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:38.354538 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:38.354505    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jun 15 06:22:39.063950 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.063903    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:39.360461 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.360430    2562 generic.go:356] "Generic (PLEG): container finished" podID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerID="500b191afafa133f032753c6fe55816b9d56e73b78a130ae00f65866f185da55" exitCode=0
Jun 15 06:22:39.360926 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.360510    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" event={"ID":"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d","Type":"ContainerDied","Data":"500b191afafa133f032753c6fe55816b9d56e73b78a130ae00f65866f185da55"}
Jun 15 06:22:39.360926 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.360849    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jun 15 06:22:39.492758 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.492729    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:22:39.686019 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.685926    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn9jm\" (UniqueName: \"kubernetes.io/projected/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-api-access-hn9jm\") pod \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") "
Jun 15 06:22:39.686019 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.686011    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls\") pod \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") "
Jun 15 06:22:39.686214 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.686066    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-rbac-proxy-sar-config\") pod \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\" (UID: \"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d\") "
Jun 15 06:22:39.686474 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.686450    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" (UID: "b87b98ed-848e-44ec-b2a8-f2b5bf5a222d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:22:39.688273 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.688248    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" (UID: "b87b98ed-848e-44ec-b2a8-f2b5bf5a222d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:22:39.688273 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.688266    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-api-access-hn9jm" (OuterVolumeSpecName: "kube-api-access-hn9jm") pod "b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" (UID: "b87b98ed-848e-44ec-b2a8-f2b5bf5a222d"). InnerVolumeSpecName "kube-api-access-hn9jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:22:39.786842 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.786803    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn9jm\" (UniqueName: \"kubernetes.io/projected/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-api-access-hn9jm\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:22:39.786842 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.786832    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:22:39.786842 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:39.786843    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:22:40.294510 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:40.294467    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jun 15 06:22:40.367020 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:40.366982    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h" event={"ID":"b87b98ed-848e-44ec-b2a8-f2b5bf5a222d","Type":"ContainerDied","Data":"0e86075a2556d7bf13837f3ee521c454a33d45c422aea79424539aeba963ddaa"}
Jun 15 06:22:40.367020 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:40.367018    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"
Jun 15 06:22:40.367568 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:40.367066    2562 scope.go:117] "RemoveContainer" containerID="770b3e983fb2fa5d19f4df7bc1a71e4ee918c3b44fd810ba15081756e4ce9477"
Jun 15 06:22:40.380285 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:40.380262    2562 scope.go:117] "RemoveContainer" containerID="500b191afafa133f032753c6fe55816b9d56e73b78a130ae00f65866f185da55"
Jun 15 06:22:40.402033 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:40.402001    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"]
Jun 15 06:22:40.418685 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:40.418652    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5c5b6-predictor-77d84cd86-hgg5h"]
Jun 15 06:22:41.489682 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:41.489648    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" path="/var/lib/kubelet/pods/b87b98ed-848e-44ec-b2a8-f2b5bf5a222d/volumes"
Jun 15 06:22:43.308642 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:43.308597    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:44.062267 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:44.062223    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:44.366495 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:44.366414    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:22:44.366955 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:44.366928    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jun 15 06:22:48.309368 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:48.309328    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:48.309860 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:48.309454    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:22:49.062684 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:49.062648    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:50.294859 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:50.294818    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jun 15 06:22:52.181963 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.181935    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:22:52.279174 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.279077    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-openshift-service-ca-bundle\") pod \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") "
Jun 15 06:22:52.279174 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.279169    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls\") pod \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\" (UID: \"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2\") "
Jun 15 06:22:52.279466 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.279441    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" (UID: "fe4fe338-5d3d-4d5d-a911-91ef4d98afc2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:22:52.281340 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.281307    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" (UID: "fe4fe338-5d3d-4d5d-a911-91ef4d98afc2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:22:52.380207 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.380166    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:22:52.380207 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.380198    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:22:52.423304 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.423272    2562 generic.go:356] "Generic (PLEG): container finished" podID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerID="7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e" exitCode=0
Jun 15 06:22:52.423515 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.423319    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"
Jun 15 06:22:52.423515 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.423360    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" event={"ID":"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2","Type":"ContainerDied","Data":"7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e"}
Jun 15 06:22:52.423515 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.423428    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg" event={"ID":"fe4fe338-5d3d-4d5d-a911-91ef4d98afc2","Type":"ContainerDied","Data":"2e5b8e3e95995aba397feae3132849f6dc865c099856308f6a50373068c1e742"}
Jun 15 06:22:52.423515 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.423450    2562 scope.go:117] "RemoveContainer" containerID="7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e"
Jun 15 06:22:52.432598 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.432519    2562 scope.go:117] "RemoveContainer" containerID="7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e"
Jun 15 06:22:52.432841 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:22:52.432822    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e\": container with ID starting with 7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e not found: ID does not exist" containerID="7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e"
Jun 15 06:22:52.432896 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.432852    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e"} err="failed to get container status \"7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e\": rpc error: code = NotFound desc = could not find container \"7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e\": container with ID starting with 7def61163a6ac6f69c0316870c2e6dff2db091fa0c00279f163f01069954414e not found: ID does not exist"
Jun 15 06:22:52.444135 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.444107    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"]
Jun 15 06:22:52.447496 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:52.447475    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-44470-76d55687fd-l8kcg"]
Jun 15 06:22:53.308949 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:53.308910    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:22:53.489161 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:53.489129    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" path="/var/lib/kubelet/pods/fe4fe338-5d3d-4d5d-a911-91ef4d98afc2/volumes"
Jun 15 06:22:54.367140 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:54.367091    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jun 15 06:22:58.308959 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:22:58.308920    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:23:00.295176 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:00.295136    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.48:8080: connect: connection refused"
Jun 15 06:23:03.309047 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:03.308992    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:23:04.367873 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:04.367831    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jun 15 06:23:06.218488 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.218463    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:23:06.292294 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.292206    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-openshift-service-ca-bundle\") pod \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") "
Jun 15 06:23:06.292294 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.292250    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls\") pod \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\" (UID: \"bd74db9f-d5d3-4f88-aaf8-6b6417c07424\") "
Jun 15 06:23:06.292649 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.292617    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bd74db9f-d5d3-4f88-aaf8-6b6417c07424" (UID: "bd74db9f-d5d3-4f88-aaf8-6b6417c07424"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:23:06.294421 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.294400    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd74db9f-d5d3-4f88-aaf8-6b6417c07424" (UID: "bd74db9f-d5d3-4f88-aaf8-6b6417c07424"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:23:06.393131 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.393096    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:23:06.393131 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.393128    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd74db9f-d5d3-4f88-aaf8-6b6417c07424-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:23:06.487921 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.487883    2562 generic.go:356] "Generic (PLEG): container finished" podID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerID="4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68" exitCode=0
Jun 15 06:23:06.487921 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.487924    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"
Jun 15 06:23:06.488239 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.487970    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" event={"ID":"bd74db9f-d5d3-4f88-aaf8-6b6417c07424","Type":"ContainerDied","Data":"4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68"}
Jun 15 06:23:06.488239 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.488025    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9" event={"ID":"bd74db9f-d5d3-4f88-aaf8-6b6417c07424","Type":"ContainerDied","Data":"bff5f7028e8a6d8178f806f193031950317e8bf71f9f908b8b1b7b2d935b4710"}
Jun 15 06:23:06.488239 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.488054    2562 scope.go:117] "RemoveContainer" containerID="4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68"
Jun 15 06:23:06.497103 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.497087    2562 scope.go:117] "RemoveContainer" containerID="4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68"
Jun 15 06:23:06.497365 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:23:06.497346    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68\": container with ID starting with 4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68 not found: ID does not exist" containerID="4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68"
Jun 15 06:23:06.497450 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.497374    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68"} err="failed to get container status \"4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68\": rpc error: code = NotFound desc = could not find container \"4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68\": container with ID starting with 4697ce420f3244d8dee135c3962c0267da913b97a060f1abd37195e1ce9f1f68 not found: ID does not exist"
Jun 15 06:23:06.508681 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.508654    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"]
Jun 15 06:23:06.512873 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:06.512853    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5c5b6-745649d7db-gtzw9"]
Jun 15 06:23:07.489457 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:07.489423    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" path="/var/lib/kubelet/pods/bd74db9f-d5d3-4f88-aaf8-6b6417c07424/volumes"
Jun 15 06:23:10.295574 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:10.295545    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:23:14.367273 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:14.367230    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.50:8080: connect: connection refused"
Jun 15 06:23:21.723460 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.723424    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"]
Jun 15 06:23:21.723944 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.723915    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kube-rbac-proxy"
Jun 15 06:23:21.723944 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.723934    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kube-rbac-proxy"
Jun 15 06:23:21.724071 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.723953    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container"
Jun 15 06:23:21.724071 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.723961    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container"
Jun 15 06:23:21.724071 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.723989    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470"
Jun 15 06:23:21.724071 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.724000    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470"
Jun 15 06:23:21.724071 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.724021    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6"
Jun 15 06:23:21.724071 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.724031    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6"
Jun 15 06:23:21.724355 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.724119    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd74db9f-d5d3-4f88-aaf8-6b6417c07424" containerName="splitter-graph-5c5b6"
Jun 15 06:23:21.724355 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.724131    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kserve-container"
Jun 15 06:23:21.724355 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.724142    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b87b98ed-848e-44ec-b2a8-f2b5bf5a222d" containerName="kube-rbac-proxy"
Jun 15 06:23:21.724355 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.724153    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe4fe338-5d3d-4d5d-a911-91ef4d98afc2" containerName="sequence-graph-44470"
Jun 15 06:23:21.727421 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.727401    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:21.729895 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.729875    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6c800-serving-cert\""
Jun 15 06:23:21.729992 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.729876    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-6c800-kube-rbac-proxy-sar-config\""
Jun 15 06:23:21.744046 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.744017    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"]
Jun 15 06:23:21.830437 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.830397    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-openshift-service-ca-bundle\") pod \"switch-graph-6c800-596ccdd7bb-g54r6\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") " pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:21.830597 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.830459    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls\") pod \"switch-graph-6c800-596ccdd7bb-g54r6\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") " pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:21.931529 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.931482    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-openshift-service-ca-bundle\") pod \"switch-graph-6c800-596ccdd7bb-g54r6\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") " pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:21.931733 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.931546    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls\") pod \"switch-graph-6c800-596ccdd7bb-g54r6\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") " pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:21.931733 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:23:21.931695    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-6c800-serving-cert: secret "switch-graph-6c800-serving-cert" not found
Jun 15 06:23:21.931855 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:23:21.931778    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls podName:54cd1c28-8bd5-4369-9d34-ebfe96c2a165 nodeName:}" failed. No retries permitted until 2026-06-15 06:23:22.431756632 +0000 UTC m=+2145.384623555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls") pod "switch-graph-6c800-596ccdd7bb-g54r6" (UID: "54cd1c28-8bd5-4369-9d34-ebfe96c2a165") : secret "switch-graph-6c800-serving-cert" not found
Jun 15 06:23:21.932295 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:21.932267    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-openshift-service-ca-bundle\") pod \"switch-graph-6c800-596ccdd7bb-g54r6\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") " pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:22.436507 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:22.436470    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls\") pod \"switch-graph-6c800-596ccdd7bb-g54r6\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") " pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:22.436729 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:23:22.436601    2562 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-6c800-serving-cert: secret "switch-graph-6c800-serving-cert" not found
Jun 15 06:23:22.436729 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:23:22.436663    2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls podName:54cd1c28-8bd5-4369-9d34-ebfe96c2a165 nodeName:}" failed. No retries permitted until 2026-06-15 06:23:23.436648874 +0000 UTC m=+2146.389515797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls") pod "switch-graph-6c800-596ccdd7bb-g54r6" (UID: "54cd1c28-8bd5-4369-9d34-ebfe96c2a165") : secret "switch-graph-6c800-serving-cert" not found
Jun 15 06:23:23.444214 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:23.444180    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls\") pod \"switch-graph-6c800-596ccdd7bb-g54r6\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") " pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:23.446761 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:23.446730    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls\") pod \"switch-graph-6c800-596ccdd7bb-g54r6\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") " pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:23.545493 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:23.545451    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:23.677454 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:23.677416    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"]
Jun 15 06:23:23.680252 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:23:23.680227    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cd1c28_8bd5_4369_9d34_ebfe96c2a165.slice/crio-c5e6278e9835812628b56fde0f70bc47fb39ad45932339416b0149879be7a125 WatchSource:0}: Error finding container c5e6278e9835812628b56fde0f70bc47fb39ad45932339416b0149879be7a125: Status 404 returned error can't find the container with id c5e6278e9835812628b56fde0f70bc47fb39ad45932339416b0149879be7a125
Jun 15 06:23:24.367543 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:24.367512    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:23:24.569603 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:24.569560    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" event={"ID":"54cd1c28-8bd5-4369-9d34-ebfe96c2a165","Type":"ContainerStarted","Data":"4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381"}
Jun 15 06:23:24.570012 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:24.569624    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" event={"ID":"54cd1c28-8bd5-4369-9d34-ebfe96c2a165","Type":"ContainerStarted","Data":"c5e6278e9835812628b56fde0f70bc47fb39ad45932339416b0149879be7a125"}
Jun 15 06:23:24.570012 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:24.569653    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:24.587645 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:24.587596    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" podStartSLOduration=3.5875810919999997 podStartE2EDuration="3.587581092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:23:24.586720426 +0000 UTC m=+2147.539587413" watchObservedRunningTime="2026-06-15 06:23:24.587581092 +0000 UTC m=+2147.540448036"
Jun 15 06:23:30.580081 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:30.580048    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:23:36.223747 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.223709    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"]
Jun 15 06:23:36.227286 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.227261    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:36.229949 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.229925    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5edeb-kube-rbac-proxy-sar-config\""
Jun 15 06:23:36.230081 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.229932    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-5edeb-serving-cert\""
Jun 15 06:23:36.235771 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.235732    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"]
Jun 15 06:23:36.355714 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.355676    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dcd8814-5e58-40db-855b-470c2872a5b7-openshift-service-ca-bundle\") pod \"splitter-graph-5edeb-655669497c-ssv4w\" (UID: \"3dcd8814-5e58-40db-855b-470c2872a5b7\") " pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:36.355914 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.355742    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dcd8814-5e58-40db-855b-470c2872a5b7-proxy-tls\") pod \"splitter-graph-5edeb-655669497c-ssv4w\" (UID: \"3dcd8814-5e58-40db-855b-470c2872a5b7\") " pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:36.456699 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.456646    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dcd8814-5e58-40db-855b-470c2872a5b7-openshift-service-ca-bundle\") pod \"splitter-graph-5edeb-655669497c-ssv4w\" (UID: \"3dcd8814-5e58-40db-855b-470c2872a5b7\") " pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:36.456891 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.456723    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dcd8814-5e58-40db-855b-470c2872a5b7-proxy-tls\") pod \"splitter-graph-5edeb-655669497c-ssv4w\" (UID: \"3dcd8814-5e58-40db-855b-470c2872a5b7\") " pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:36.457303 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.457279    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dcd8814-5e58-40db-855b-470c2872a5b7-openshift-service-ca-bundle\") pod \"splitter-graph-5edeb-655669497c-ssv4w\" (UID: \"3dcd8814-5e58-40db-855b-470c2872a5b7\") " pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:36.459255 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.459238    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dcd8814-5e58-40db-855b-470c2872a5b7-proxy-tls\") pod \"splitter-graph-5edeb-655669497c-ssv4w\" (UID: \"3dcd8814-5e58-40db-855b-470c2872a5b7\") " pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:36.540555 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.540476    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:36.673733 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:36.673703    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"]
Jun 15 06:23:36.676897 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:23:36.676866    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dcd8814_5e58_40db_855b_470c2872a5b7.slice/crio-87574a40b58cf084b4296225c6ef423935d4bfcbc429ec59ebce8d4b95fd6953 WatchSource:0}: Error finding container 87574a40b58cf084b4296225c6ef423935d4bfcbc429ec59ebce8d4b95fd6953: Status 404 returned error can't find the container with id 87574a40b58cf084b4296225c6ef423935d4bfcbc429ec59ebce8d4b95fd6953
Jun 15 06:23:37.631586 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:37.631544    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" event={"ID":"3dcd8814-5e58-40db-855b-470c2872a5b7","Type":"ContainerStarted","Data":"34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae"}
Jun 15 06:23:37.631926 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:37.631602    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" event={"ID":"3dcd8814-5e58-40db-855b-470c2872a5b7","Type":"ContainerStarted","Data":"87574a40b58cf084b4296225c6ef423935d4bfcbc429ec59ebce8d4b95fd6953"}
Jun 15 06:23:37.631926 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:37.631634    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:23:37.650122 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:37.650073    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" podStartSLOduration=1.6500598530000001 podStartE2EDuration="1.650059853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:23:37.648527742 +0000 UTC m=+2160.601394689" watchObservedRunningTime="2026-06-15 06:23:37.650059853 +0000 UTC m=+2160.602926797"
Jun 15 06:23:43.641489 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:23:43.641456    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:27:37.627510 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:27:37.627484    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:27:37.633718 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:27:37.633696    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:31:50.922005 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:50.921969    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"]
Jun 15 06:31:50.922632 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:50.922172    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb" containerID="cri-o://34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae" gracePeriod=30
Jun 15 06:31:51.026759 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:51.026718    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"]
Jun 15 06:31:51.027056 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:51.027004    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container" containerID="cri-o://6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc" gracePeriod=30
Jun 15 06:31:51.027216 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:51.027069    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kube-rbac-proxy" containerID="cri-o://64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db" gracePeriod=30
Jun 15 06:31:51.808442 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:51.808408    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerID="64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db" exitCode=2
Jun 15 06:31:51.808620 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:51.808482    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" event={"ID":"d0fec994-cf65-43fc-b0a1-e5db0b83cc94","Type":"ContainerDied","Data":"64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db"}
Jun 15 06:31:53.639839 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:53.639799    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:31:54.177053 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.177021    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:31:54.289284 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.289189    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-rbac-proxy-sar-config\") pod \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") "
Jun 15 06:31:54.289284 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.289236    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkwfq\" (UniqueName: \"kubernetes.io/projected/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-api-access-pkwfq\") pod \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") "
Jun 15 06:31:54.289284 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.289267    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls\") pod \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\" (UID: \"d0fec994-cf65-43fc-b0a1-e5db0b83cc94\") "
Jun 15 06:31:54.289685 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.289646    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0fec994-cf65-43fc-b0a1-e5db0b83cc94" (UID: "d0fec994-cf65-43fc-b0a1-e5db0b83cc94"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:31:54.291562 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.291539    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0fec994-cf65-43fc-b0a1-e5db0b83cc94" (UID: "d0fec994-cf65-43fc-b0a1-e5db0b83cc94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:31:54.291639 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.291588    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-api-access-pkwfq" (OuterVolumeSpecName: "kube-api-access-pkwfq") pod "d0fec994-cf65-43fc-b0a1-e5db0b83cc94" (UID: "d0fec994-cf65-43fc-b0a1-e5db0b83cc94"). InnerVolumeSpecName "kube-api-access-pkwfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:31:54.390538 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.390496    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:31:54.390538 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.390534    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:31:54.390538 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.390547    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkwfq\" (UniqueName: \"kubernetes.io/projected/d0fec994-cf65-43fc-b0a1-e5db0b83cc94-kube-api-access-pkwfq\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:31:54.823473 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.823436    2562 generic.go:356] "Generic (PLEG): container finished" podID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerID="6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc" exitCode=0
Jun 15 06:31:54.823907 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.823491    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"
Jun 15 06:31:54.823907 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.823519    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" event={"ID":"d0fec994-cf65-43fc-b0a1-e5db0b83cc94","Type":"ContainerDied","Data":"6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc"}
Jun 15 06:31:54.823907 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.823586    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9" event={"ID":"d0fec994-cf65-43fc-b0a1-e5db0b83cc94","Type":"ContainerDied","Data":"46a9c05e457c0ae41099fb442795ddb1ca529a196ed9b5142db4a11eb9a923e5"}
Jun 15 06:31:54.823907 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.823627    2562 scope.go:117] "RemoveContainer" containerID="64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db"
Jun 15 06:31:54.835400 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.835362    2562 scope.go:117] "RemoveContainer" containerID="6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc"
Jun 15 06:31:54.843307 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.843286    2562 scope.go:117] "RemoveContainer" containerID="64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db"
Jun 15 06:31:54.843603 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:31:54.843584    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db\": container with ID starting with 64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db not found: ID does not exist" containerID="64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db"
Jun 15 06:31:54.843646 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.843613    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db"} err="failed to get container status \"64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db\": rpc error: code = NotFound desc = could not find container \"64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db\": container with ID starting with 64771040341d27ba03cc3bef50af49f7b295ec669abdbf81a943d675822a67db not found: ID does not exist"
Jun 15 06:31:54.843646 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.843635    2562 scope.go:117] "RemoveContainer" containerID="6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc"
Jun 15 06:31:54.843900 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:31:54.843883    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc\": container with ID starting with 6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc not found: ID does not exist" containerID="6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc"
Jun 15 06:31:54.843949 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.843906    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc"} err="failed to get container status \"6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc\": rpc error: code = NotFound desc = could not find container \"6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc\": container with ID starting with 6c23b1960e298404a61d6fdd91de0e16c6a6302ecbe88c6bdf064bb44d85b0cc not found: ID does not exist"
Jun 15 06:31:54.849212 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.849188    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"]
Jun 15 06:31:54.855622 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:54.855601    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5edeb-predictor-869ffb65f4-m57w9"]
Jun 15 06:31:55.489024 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:55.488994    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" path="/var/lib/kubelet/pods/d0fec994-cf65-43fc-b0a1-e5db0b83cc94/volumes"
Jun 15 06:31:58.640067 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:31:58.640025    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:32:03.640297 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:03.640259    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:32:03.640743 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:03.640367    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:32:08.639973 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:08.639930    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:32:13.640204 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:13.640155    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:32:18.639733 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:18.639690    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:32:21.073584 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.073556    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:32:21.216666 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.216571    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dcd8814-5e58-40db-855b-470c2872a5b7-openshift-service-ca-bundle\") pod \"3dcd8814-5e58-40db-855b-470c2872a5b7\" (UID: \"3dcd8814-5e58-40db-855b-470c2872a5b7\") "
Jun 15 06:32:21.216666 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.216611    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dcd8814-5e58-40db-855b-470c2872a5b7-proxy-tls\") pod \"3dcd8814-5e58-40db-855b-470c2872a5b7\" (UID: \"3dcd8814-5e58-40db-855b-470c2872a5b7\") "
Jun 15 06:32:21.216972 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.216947    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dcd8814-5e58-40db-855b-470c2872a5b7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "3dcd8814-5e58-40db-855b-470c2872a5b7" (UID: "3dcd8814-5e58-40db-855b-470c2872a5b7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:32:21.218829 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.218809    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dcd8814-5e58-40db-855b-470c2872a5b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3dcd8814-5e58-40db-855b-470c2872a5b7" (UID: "3dcd8814-5e58-40db-855b-470c2872a5b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:32:21.317396 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.317347    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dcd8814-5e58-40db-855b-470c2872a5b7-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:32:21.317396 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.317376    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3dcd8814-5e58-40db-855b-470c2872a5b7-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:32:21.943365 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.943280    2562 generic.go:356] "Generic (PLEG): container finished" podID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerID="34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae" exitCode=0
Jun 15 06:32:21.943365 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.943325    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"
Jun 15 06:32:21.943648 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.943396    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" event={"ID":"3dcd8814-5e58-40db-855b-470c2872a5b7","Type":"ContainerDied","Data":"34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae"}
Jun 15 06:32:21.943648 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.943454    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w" event={"ID":"3dcd8814-5e58-40db-855b-470c2872a5b7","Type":"ContainerDied","Data":"87574a40b58cf084b4296225c6ef423935d4bfcbc429ec59ebce8d4b95fd6953"}
Jun 15 06:32:21.943648 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.943477    2562 scope.go:117] "RemoveContainer" containerID="34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae"
Jun 15 06:32:21.951938 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.951919    2562 scope.go:117] "RemoveContainer" containerID="34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae"
Jun 15 06:32:21.952224 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:32:21.952205    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae\": container with ID starting with 34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae not found: ID does not exist" containerID="34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae"
Jun 15 06:32:21.952272 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.952233    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae"} err="failed to get container status \"34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae\": rpc error: code = NotFound desc = could not find container \"34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae\": container with ID starting with 34421a22488332a48cbd971d5e11fbb29385ec3de83f2dec8b0470cec0718cae not found: ID does not exist"
Jun 15 06:32:21.959235 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.959206    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"]
Jun 15 06:32:21.962457 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:21.962426    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-5edeb-655669497c-ssv4w"]
Jun 15 06:32:23.488998 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:23.488965    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" path="/var/lib/kubelet/pods/3dcd8814-5e58-40db-855b-470c2872a5b7/volumes"
Jun 15 06:32:37.651107 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:37.651078    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:32:37.658755 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:32:37.658733    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:37:37.675061 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:37:37.675030    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:37:37.690291 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:37:37.690269    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:39:41.151134 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:41.151097    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"]
Jun 15 06:39:41.151671 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:41.151332    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800" containerID="cri-o://4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381" gracePeriod=30
Jun 15 06:39:41.287285 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:41.287239    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"]
Jun 15 06:39:41.287576 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:41.287534    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container" containerID="cri-o://10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888" gracePeriod=30
Jun 15 06:39:41.287673 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:41.287622    2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kube-rbac-proxy" containerID="cri-o://fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300" gracePeriod=30
Jun 15 06:39:41.836710 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:41.836675    2562 generic.go:356] "Generic (PLEG): container finished" podID="71fd1f5e-b994-4010-90dd-b1158035917a" containerID="fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300" exitCode=2
Jun 15 06:39:41.836879 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:41.836744    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" event={"ID":"71fd1f5e-b994-4010-90dd-b1158035917a","Type":"ContainerDied","Data":"fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300"}
Jun 15 06:39:44.133416 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.133361    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:39:44.194069 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.194025    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2pwp\" (UniqueName: \"kubernetes.io/projected/71fd1f5e-b994-4010-90dd-b1158035917a-kube-api-access-x2pwp\") pod \"71fd1f5e-b994-4010-90dd-b1158035917a\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") "
Jun 15 06:39:44.194069 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.194079    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls\") pod \"71fd1f5e-b994-4010-90dd-b1158035917a\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") "
Jun 15 06:39:44.194333 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.194108    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71fd1f5e-b994-4010-90dd-b1158035917a-kube-rbac-proxy-sar-config\") pod \"71fd1f5e-b994-4010-90dd-b1158035917a\" (UID: \"71fd1f5e-b994-4010-90dd-b1158035917a\") "
Jun 15 06:39:44.194521 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.194496    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71fd1f5e-b994-4010-90dd-b1158035917a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71fd1f5e-b994-4010-90dd-b1158035917a" (UID: "71fd1f5e-b994-4010-90dd-b1158035917a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:39:44.196334 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.196307    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71fd1f5e-b994-4010-90dd-b1158035917a" (UID: "71fd1f5e-b994-4010-90dd-b1158035917a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:39:44.196472 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.196356    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fd1f5e-b994-4010-90dd-b1158035917a-kube-api-access-x2pwp" (OuterVolumeSpecName: "kube-api-access-x2pwp") pod "71fd1f5e-b994-4010-90dd-b1158035917a" (UID: "71fd1f5e-b994-4010-90dd-b1158035917a"). InnerVolumeSpecName "kube-api-access-x2pwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:39:44.295170 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.295054    2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2pwp\" (UniqueName: \"kubernetes.io/projected/71fd1f5e-b994-4010-90dd-b1158035917a-kube-api-access-x2pwp\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:39:44.295170 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.295103    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71fd1f5e-b994-4010-90dd-b1158035917a-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:39:44.295170 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.295114    2562 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71fd1f5e-b994-4010-90dd-b1158035917a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:39:44.851244 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.851212    2562 generic.go:356] "Generic (PLEG): container finished" podID="71fd1f5e-b994-4010-90dd-b1158035917a" containerID="10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888" exitCode=0
Jun 15 06:39:44.851476 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.851268    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"
Jun 15 06:39:44.851476 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.851265    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" event={"ID":"71fd1f5e-b994-4010-90dd-b1158035917a","Type":"ContainerDied","Data":"10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888"}
Jun 15 06:39:44.851476 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.851425    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh" event={"ID":"71fd1f5e-b994-4010-90dd-b1158035917a","Type":"ContainerDied","Data":"52b1f6a29b58da23537e03227f8bcb194b453f8e54daf85e281163c94d27900a"}
Jun 15 06:39:44.851476 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.851455    2562 scope.go:117] "RemoveContainer" containerID="fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300"
Jun 15 06:39:44.860276 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.860258    2562 scope.go:117] "RemoveContainer" containerID="10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888"
Jun 15 06:39:44.867900 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.867880    2562 scope.go:117] "RemoveContainer" containerID="fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300"
Jun 15 06:39:44.868200 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:39:44.868181    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300\": container with ID starting with fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300 not found: ID does not exist" containerID="fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300"
Jun 15 06:39:44.868290 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.868209    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300"} err="failed to get container status \"fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300\": rpc error: code = NotFound desc = could not find container \"fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300\": container with ID starting with fb99c16e7618b6653bcb974542f32079c8173ddc1297c2125017542938cf8300 not found: ID does not exist"
Jun 15 06:39:44.868290 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.868227    2562 scope.go:117] "RemoveContainer" containerID="10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888"
Jun 15 06:39:44.868482 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:39:44.868460    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888\": container with ID starting with 10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888 not found: ID does not exist" containerID="10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888"
Jun 15 06:39:44.868536 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.868494    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888"} err="failed to get container status \"10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888\": rpc error: code = NotFound desc = could not find container \"10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888\": container with ID starting with 10ad1b9bfba08a35f751036b1028a6174cefd5c569439a77aa7f34b0eee26888 not found: ID does not exist"
Jun 15 06:39:44.873127 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.873104    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"]
Jun 15 06:39:44.876661 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:44.876639    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-6c800-predictor-866b9bb7cd-p55wh"]
Jun 15 06:39:45.489644 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:45.489603    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" path="/var/lib/kubelet/pods/71fd1f5e-b994-4010-90dd-b1158035917a/volumes"
Jun 15 06:39:45.577543 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:45.577502    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:39:50.577668 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:50.577628    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:39:55.577608 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:55.577515    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:39:55.577988 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:55.577674    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:39:58.384201 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:58.384172    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:39:59.239255 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:39:59.239200    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:00.080849 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:00.080822    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:00.577611 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:00.577569    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:40:00.921572 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:00.921488    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:01.758372 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:01.758333    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:02.606426 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:02.606396    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:03.465503 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:03.465472    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:04.313637 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:04.313606    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:05.198075 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:05.198033    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:05.577544 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:05.577507    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:40:06.006144 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:06.006098    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:06.824044 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:06.823996    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:07.649847 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:07.649815    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-6c800-596ccdd7bb-g54r6_54cd1c28-8bd5-4369-9d34-ebfe96c2a165/switch-graph-6c800/0.log"
Jun 15 06:40:10.577433 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:10.577364    2562 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:40:11.303857 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.303831    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:40:11.430003 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.429918    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls\") pod \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") "
Jun 15 06:40:11.430003 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.429968    2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-openshift-service-ca-bundle\") pod \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\" (UID: \"54cd1c28-8bd5-4369-9d34-ebfe96c2a165\") "
Jun 15 06:40:11.430419 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.430361    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "54cd1c28-8bd5-4369-9d34-ebfe96c2a165" (UID: "54cd1c28-8bd5-4369-9d34-ebfe96c2a165"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:40:11.432149 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.432121    2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "54cd1c28-8bd5-4369-9d34-ebfe96c2a165" (UID: "54cd1c28-8bd5-4369-9d34-ebfe96c2a165"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:40:11.530956 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.530927    2562 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-proxy-tls\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:40:11.530956 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.530954    2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54cd1c28-8bd5-4369-9d34-ebfe96c2a165-openshift-service-ca-bundle\") on node \"ip-10-0-133-111.ec2.internal\" DevicePath \"\""
Jun 15 06:40:11.968278 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.968245    2562 generic.go:356] "Generic (PLEG): container finished" podID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerID="4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381" exitCode=0
Jun 15 06:40:11.968795 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.968296    2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"
Jun 15 06:40:11.968795 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.968340    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" event={"ID":"54cd1c28-8bd5-4369-9d34-ebfe96c2a165","Type":"ContainerDied","Data":"4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381"}
Jun 15 06:40:11.968795 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.968423    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6" event={"ID":"54cd1c28-8bd5-4369-9d34-ebfe96c2a165","Type":"ContainerDied","Data":"c5e6278e9835812628b56fde0f70bc47fb39ad45932339416b0149879be7a125"}
Jun 15 06:40:11.968795 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.968452    2562 scope.go:117] "RemoveContainer" containerID="4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381"
Jun 15 06:40:11.976438 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.976416    2562 scope.go:117] "RemoveContainer" containerID="4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381"
Jun 15 06:40:11.976717 ip-10-0-133-111 kubenswrapper[2562]: E0615 06:40:11.976695    2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381\": container with ID starting with 4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381 not found: ID does not exist" containerID="4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381"
Jun 15 06:40:11.976765 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.976726    2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381"} err="failed to get container status \"4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381\": rpc error: code = NotFound desc = could not find container \"4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381\": container with ID starting with 4ccb6e5d8f7ec5cb1909c4aed9706700b91e90fd8b497592c43c7eef14de2381 not found: ID does not exist"
Jun 15 06:40:11.985898 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.985874    2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"]
Jun 15 06:40:11.991212 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:11.991187    2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-6c800-596ccdd7bb-g54r6"]
Jun 15 06:40:13.489639 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:13.489598    2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" path="/var/lib/kubelet/pods/54cd1c28-8bd5-4369-9d34-ebfe96c2a165/volumes"
Jun 15 06:40:16.372665 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:16.372638    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6hzm9_937dcb5e-3893-4ff2-a2f6-a3330a232700/global-pull-secret-syncer/0.log"
Jun 15 06:40:16.481343 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:16.481312    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gbjb7_14e05b24-a058-4fef-b061-38cb4453e6c2/konnectivity-agent/0.log"
Jun 15 06:40:16.593354 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:16.593317    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-111.ec2.internal_08d00837dae608698233b66e665078bb/haproxy/0.log"
Jun 15 06:40:21.125003 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:21.124972    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-f9d7df769-d45f6_0bde2cab-f15d-4308-b2e7-5c02547676a3/cluster-monitoring-operator/0.log"
Jun 15 06:40:21.406945 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:21.406846    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w9mgx_98905fcb-4803-43a5-a34a-00ae8a407f70/node-exporter/0.log"
Jun 15 06:40:21.430064 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:21.430037    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w9mgx_98905fcb-4803-43a5-a34a-00ae8a407f70/kube-rbac-proxy/0.log"
Jun 15 06:40:21.451584 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:21.451558    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w9mgx_98905fcb-4803-43a5-a34a-00ae8a407f70/init-textfile/0.log"
Jun 15 06:40:22.031118 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:22.031057    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5cf667749c-xrrb7_a1aa4784-b609-4da2-b96e-479cd17ab850/prometheus-operator-admission-webhook/0.log"
Jun 15 06:40:24.336063 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.336031    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bfc59d58d-txdlv_683f9fef-7c97-4f97-8f68-d8ece1476cdc/console/0.log"
Jun 15 06:40:24.395822 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.395790    2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"]
Jun 15 06:40:24.396094 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396082    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kube-rbac-proxy"
Jun 15 06:40:24.396138 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396097    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kube-rbac-proxy"
Jun 15 06:40:24.396138 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396108    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container"
Jun 15 06:40:24.396138 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396114    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container"
Jun 15 06:40:24.396138 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396123    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kube-rbac-proxy"
Jun 15 06:40:24.396138 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396129    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kube-rbac-proxy"
Jun 15 06:40:24.396138 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396138    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396144    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396153    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396158    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396166    2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396171    2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396218    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kserve-container"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396228    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kserve-container"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396236    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0fec994-cf65-43fc-b0a1-e5db0b83cc94" containerName="kube-rbac-proxy"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396241    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="71fd1f5e-b994-4010-90dd-b1158035917a" containerName="kube-rbac-proxy"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396248    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="54cd1c28-8bd5-4369-9d34-ebfe96c2a165" containerName="switch-graph-6c800"
Jun 15 06:40:24.396320 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.396253    2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dcd8814-5e58-40db-855b-470c2872a5b7" containerName="splitter-graph-5edeb"
Jun 15 06:40:24.400822 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.400782    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.403697 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.403673    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8rxx\"/\"kube-root-ca.crt\""
Jun 15 06:40:24.404794 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.404771    2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8rxx\"/\"default-dockercfg-t4kgc\""
Jun 15 06:40:24.404794 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.404784    2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8rxx\"/\"openshift-service-ca.crt\""
Jun 15 06:40:24.406464 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.406440    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"]
Jun 15 06:40:24.427079 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.427046    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-lib-modules\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.427079 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.427079    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-sys\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.427276 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.427105    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-proc\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.427276 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.427183    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4x2m\" (UniqueName: \"kubernetes.io/projected/f16478e5-b133-4bf9-97a0-378703f16f98-kube-api-access-q4x2m\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.427276 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.427252    2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-podres\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528161 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528124    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-podres\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528161 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528165    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-lib-modules\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528433 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528181    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-sys\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528433 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528211    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-proc\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528433 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528256    2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4x2m\" (UniqueName: \"kubernetes.io/projected/f16478e5-b133-4bf9-97a0-378703f16f98-kube-api-access-q4x2m\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528433 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528277    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-sys\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528433 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528291    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-podres\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528433 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528313    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-lib-modules\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.528433 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.528345    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f16478e5-b133-4bf9-97a0-378703f16f98-proc\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.538723 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.538695    2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4x2m\" (UniqueName: \"kubernetes.io/projected/f16478e5-b133-4bf9-97a0-378703f16f98-kube-api-access-q4x2m\") pod \"perf-node-gather-daemonset-b4bmn\" (UID: \"f16478e5-b133-4bf9-97a0-378703f16f98\") " pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.713092 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.713000    2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:24.852135 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.852085    2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"]
Jun 15 06:40:24.854560 ip-10-0-133-111 kubenswrapper[2562]: W0615 06:40:24.854534    2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf16478e5_b133_4bf9_97a0_378703f16f98.slice/crio-cf12f6be252ac009b47a58fc4c6d2fb621523414d80e6a33bf1e7c84e6560dbc WatchSource:0}: Error finding container cf12f6be252ac009b47a58fc4c6d2fb621523414d80e6a33bf1e7c84e6560dbc: Status 404 returned error can't find the container with id cf12f6be252ac009b47a58fc4c6d2fb621523414d80e6a33bf1e7c84e6560dbc
Jun 15 06:40:24.956268 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:24.956236    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-bd4ccd799-nbhgb_7e8cd637-4880-4ab6-9585-d58abfef84c1/volume-data-source-validator/0.log"
Jun 15 06:40:25.029401 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:25.029345    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn" event={"ID":"f16478e5-b133-4bf9-97a0-378703f16f98","Type":"ContainerStarted","Data":"7da538d3bbaf2eebe537aabccbec7e0a01fe9e0af1d679fb368615825f14447c"}
Jun 15 06:40:25.029572 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:25.029411    2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn" event={"ID":"f16478e5-b133-4bf9-97a0-378703f16f98","Type":"ContainerStarted","Data":"cf12f6be252ac009b47a58fc4c6d2fb621523414d80e6a33bf1e7c84e6560dbc"}
Jun 15 06:40:25.029572 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:25.029555    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:25.048550 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:25.048502    2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn" podStartSLOduration=1.048487638 podStartE2EDuration="1.048487638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 06:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:40:25.046953096 +0000 UTC m=+3167.999820041" watchObservedRunningTime="2026-06-15 06:40:25.048487638 +0000 UTC m=+3168.001354583"
Jun 15 06:40:25.971084 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:25.971056    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zwqwd_30915d28-dba8-4a63-a4f6-367fe7a59f4a/dns/0.log"
Jun 15 06:40:25.993958 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:25.993934    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zwqwd_30915d28-dba8-4a63-a4f6-367fe7a59f4a/kube-rbac-proxy/0.log"
Jun 15 06:40:26.019162 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:26.019134    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dzgqh_a0a550a5-e9e9-4c12-b69d-8ae4a696ffd7/dns-node-resolver/0.log"
Jun 15 06:40:26.544223 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:26.544180    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-65f46bd878-q4dd2_a330b581-0200-4f0e-8eb7-683a8e06cb04/registry/0.log"
Jun 15 06:40:26.564484 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:26.564445    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4xgl2_48938dcf-66f0-4920-be0c-f3735090e6c2/node-ca/0.log"
Jun 15 06:40:27.821308 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:27.821276    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5tnsl_f4c07b8d-4964-4899-a3ff-5f9b0fa13728/serve-healthcheck-canary/0.log"
Jun 15 06:40:28.266666 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:28.266639    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvd67_5861fb58-e417-4d6d-8720-81f66fbca9d4/kube-rbac-proxy/0.log"
Jun 15 06:40:28.287243 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:28.287216    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvd67_5861fb58-e417-4d6d-8720-81f66fbca9d4/exporter/0.log"
Jun 15 06:40:28.309058 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:28.309032    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvd67_5861fb58-e417-4d6d-8720-81f66fbca9d4/extractor/0.log"
Jun 15 06:40:30.397115 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:30.397074    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6c7654bd99-d26gb_360b7d2c-63c7-4cb1-b793-d0b7d942e733/manager/0.log"
Jun 15 06:40:30.442757 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:30.442733    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-n79ql_85dfd84e-d031-49e5-8f5e-e89e4cdc2ef5/server/0.log"
Jun 15 06:40:30.689623 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:30.689512    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-j6zn7_c41efd46-aacd-446f-9067-99f9e8b2b1fc/seaweedfs/0.log"
Jun 15 06:40:31.043930 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:31.043899    2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8rxx/perf-node-gather-daemonset-b4bmn"
Jun 15 06:40:34.833330 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:34.833291    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d4bcd4867-9wjqm_9bab946d-a7ea-473e-9e9e-00389436e017/migrator/0.log"
Jun 15 06:40:34.857035 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:34.857011    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d4bcd4867-9wjqm_9bab946d-a7ea-473e-9e9e-00389436e017/graceful-termination/0.log"
Jun 15 06:40:36.455416 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.455368    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kmskc_61f89985-9448-4de8-b773-dbeb59284d44/kube-multus-additional-cni-plugins/0.log"
Jun 15 06:40:36.476737 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.476712    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kmskc_61f89985-9448-4de8-b773-dbeb59284d44/egress-router-binary-copy/0.log"
Jun 15 06:40:36.497562 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.497536    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kmskc_61f89985-9448-4de8-b773-dbeb59284d44/cni-plugins/0.log"
Jun 15 06:40:36.517939 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.517917    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kmskc_61f89985-9448-4de8-b773-dbeb59284d44/bond-cni-plugin/0.log"
Jun 15 06:40:36.540314 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.540292    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kmskc_61f89985-9448-4de8-b773-dbeb59284d44/routeoverride-cni/0.log"
Jun 15 06:40:36.560515 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.560488    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kmskc_61f89985-9448-4de8-b773-dbeb59284d44/whereabouts-cni-bincopy/0.log"
Jun 15 06:40:36.581891 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.581868    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kmskc_61f89985-9448-4de8-b773-dbeb59284d44/whereabouts-cni/0.log"
Jun 15 06:40:36.614918 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.614882    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cl972_ace5f3e3-d61e-41cb-ba3f-e72abf8367b4/kube-multus/0.log"
Jun 15 06:40:36.740860 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.740830    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cplw2_7139966e-e862-401e-bf2b-f13f4f334356/network-metrics-daemon/0.log"
Jun 15 06:40:36.761297 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:36.761268    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-cplw2_7139966e-e862-401e-bf2b-f13f4f334356/kube-rbac-proxy/0.log"
Jun 15 06:40:38.034153 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.034126    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-controller/0.log"
Jun 15 06:40:38.052632 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.052595    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/1.log"
Jun 15 06:40:38.052777 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.052687    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovn-acl-logging/0.log"
Jun 15 06:40:38.073505 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.073475    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/kube-rbac-proxy-node/0.log"
Jun 15 06:40:38.094807 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.094740    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/kube-rbac-proxy-ovn-metrics/0.log"
Jun 15 06:40:38.113479 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.113452    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/northd/0.log"
Jun 15 06:40:38.135205 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.135169    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/nbdb/0.log"
Jun 15 06:40:38.158510 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.158487    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/sbdb/0.log"
Jun 15 06:40:38.205037 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:38.204991    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rm8q8_441e22c2-c489-4c53-aa0f-4d96208b16ff/ovnkube-controller/0.log"
Jun 15 06:40:39.204246 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:39.204213    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6947749cfc-6cq7b_8ee9b7bb-801c-4c8f-83d0-82872c001eac/check-endpoints/0.log"
Jun 15 06:40:39.229359 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:39.229336    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-984wm_c7c76571-ac25-4149-93e0-1c27217e11d1/network-check-target-container/0.log"
Jun 15 06:40:40.299112 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:40.299083    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-hrdzf_9dd6fe0d-463d-4df8-a826-66922919a981/iptables-alerter/0.log"
Jun 15 06:40:41.032352 ip-10-0-133-111 kubenswrapper[2562]: I0615 06:40:41.032325    2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-7vb9c_f042f86e-21bb-4d7a-9141-e3fce2adadb3/tuned/0.log"