May 26 15:58:22.255568 ip-10-0-133-131 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
May 26 15:58:22.255577 ip-10-0-133-131 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
May 26 15:58:22.255584 ip-10-0-133-131 systemd[1]: kubelet.service: Failed with result 'resources'.
May 26 15:58:22.255792 ip-10-0-133-131 systemd[1]: Failed to start Kubernetes Kubelet.
May 26 15:58:33.265162 ip-10-0-133-131 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
May 26 15:58:33.265178 ip-10-0-133-131 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e2f28969389548879d6d4ba8a1f86c2e --
May 26 16:00:53.005173 ip-10-0-133-131 systemd[1]: Starting Kubernetes Kubelet...
May 26 16:00:53.463186 ip-10-0-133-131 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:53.463186 ip-10-0-133-131 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 26 16:00:53.463186 ip-10-0-133-131 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:53.463186 ip-10-0-133-131 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 26 16:00:53.463186 ip-10-0-133-131 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 26 16:00:53.464928 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.464818    2561 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 26 16:00:53.469232 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469210    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:53.469232 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469226    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:53.469232 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469232    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:53.469232 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469236    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469241    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469245    2561 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469249    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469252    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469256    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469261    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469264    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469267    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469271    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469275    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469279    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469283    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469286    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469290    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469294    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469297    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469301    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469305    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469308    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:53.469470 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469312    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469316    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469320    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469324    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469330    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469333    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469337    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469344    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469350    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469354    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469359    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469363    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469368    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469372    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469377    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469392    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469398    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469403    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469407    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:53.470295 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469412    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469416    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469420    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469424    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469430    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469434    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469438    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469443    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469447    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469451    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469456    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469460    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469465    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469469    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469474    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469478    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469482    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469487    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469491    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469495    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:53.471123 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469499    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469507    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469511    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469516    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469520    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469524    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469528    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469533    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469538    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469542    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469546    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469550    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469554    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469559    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469563    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469567    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469572    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469575    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469579    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:53.471976 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469584    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469588    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469592    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469596    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.469600    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470296    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470305    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470309    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470313    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470319    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470332    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470337    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470342    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470346    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470350    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470354    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470358    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470362    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470366    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470370    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:53.472655 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470377    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470381    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470386    2561 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470390    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470394    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470398    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470403    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470407    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470415    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470420    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470424    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470428    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470432    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470436    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470440    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470446    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470450    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470454    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470458    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470462    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:53.473302 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470466    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470471    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470474    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470486    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470491    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470494    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470498    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470503    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470506    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470511    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470515    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470519    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470524    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470528    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470536    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470541    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470546    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470551    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470555    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:53.473928 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470560    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470564    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470568    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470572    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470578    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470582    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470586    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470590    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470594    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470598    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470602    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470606    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470610    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470615    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470620    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470625    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470629    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470642    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470646    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:53.474402 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470651    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470654    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470658    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470662    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470667    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470671    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470676    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470680    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470684    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470688    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470693    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470697    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.470701    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470807    2561 flags.go:64] FLAG: --address="0.0.0.0"
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470818    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470833    2561 flags.go:64] FLAG: --anonymous-auth="true"
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470841    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470848    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470854    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470861    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470869    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 26 16:00:53.474984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470875    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470880    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470904    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470910    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470915    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470920    2561 flags.go:64] FLAG: --cgroup-root=""
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470925    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470929    2561 flags.go:64] FLAG: --client-ca-file=""
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470934    2561 flags.go:64] FLAG: --cloud-config=""
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470939    2561 flags.go:64] FLAG: --cloud-provider="external"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470955    2561 flags.go:64] FLAG: --cluster-dns="[]"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470965    2561 flags.go:64] FLAG: --cluster-domain=""
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470969    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470975    2561 flags.go:64] FLAG: --config-dir=""
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470980    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470985    2561 flags.go:64] FLAG: --container-log-max-files="5"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470992    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.470997    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471002    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471008    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471012    2561 flags.go:64] FLAG: --contention-profiling="false"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471017    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471022    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471028    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471032    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 26 16:00:53.475495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471039    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471044    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471050    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471054    2561 flags.go:64] FLAG: --enable-load-reader="false"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471060    2561 flags.go:64] FLAG: --enable-server="true"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471065    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471076    2561 flags.go:64] FLAG: --event-burst="100"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471081    2561 flags.go:64] FLAG: --event-qps="50"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471086    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471091    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471095    2561 flags.go:64] FLAG: --eviction-hard=""
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471101    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471106    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471111    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471117    2561 flags.go:64] FLAG: --eviction-soft=""
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471121    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471126    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471130    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471143    2561 flags.go:64] FLAG: --experimental-mounter-path=""
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471148    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471153    2561 flags.go:64] FLAG: --fail-swap-on="true"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471158    2561 flags.go:64] FLAG: --feature-gates=""
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471165    2561 flags.go:64] FLAG: --file-check-frequency="20s"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471175    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471180    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 26 16:00:53.476135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471185    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471191    2561 flags.go:64] FLAG: --healthz-port="10248"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471196    2561 flags.go:64] FLAG: --help="false"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471201    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-133-131.ec2.internal"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471206    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471212    2561 flags.go:64] FLAG: --http-check-frequency="20s"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471216    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471225    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471231    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471236    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471240    2561 flags.go:64] FLAG: --image-service-endpoint=""
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471245    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471250    2561 flags.go:64] FLAG: --kube-api-burst="100"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471255    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471260    2561 flags.go:64] FLAG: --kube-api-qps="50"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471265    2561 flags.go:64] FLAG: --kube-reserved=""
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471270    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471274    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471280    2561 flags.go:64] FLAG: --kubelet-cgroups=""
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471285    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471290    2561 flags.go:64] FLAG: --lock-file=""
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471295    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471299    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471304    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 26 16:00:53.476746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471313    2561 flags.go:64] FLAG: --log-json-split-stream="false"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471318    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471330    2561 flags.go:64] FLAG: --log-text-split-stream="false"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471335    2561 flags.go:64] FLAG: --logging-format="text"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471340    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471345    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471352    2561 flags.go:64] FLAG: --manifest-url=""
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471356    2561 flags.go:64] FLAG: --manifest-url-header=""
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471363    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471368    2561 flags.go:64] FLAG: --max-open-files="1000000"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471375    2561 flags.go:64] FLAG: --max-pods="110"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471380    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471385    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471389    2561 flags.go:64] FLAG: --memory-manager-policy="None"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471394    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471401    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471406    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471411    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471423    2561 flags.go:64] FLAG: --node-status-max-images="50"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471428    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471433    2561 flags.go:64] FLAG: --oom-score-adj="-999"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471438    2561 flags.go:64] FLAG: --pod-cidr=""
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471443    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 26 16:00:53.477419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471456    2561 flags.go:64] FLAG: --pod-manifest-path=""
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471461    2561 flags.go:64] FLAG: --pod-max-pids="-1"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471466    2561 flags.go:64] FLAG: --pods-per-core="0"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471470    2561 flags.go:64] FLAG: --port="10250"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471475    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471480    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0357bc9d948985d19"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471486    2561 flags.go:64] FLAG: --qos-reserved=""
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471492    2561 flags.go:64] FLAG: --read-only-port="10255"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471496    2561 flags.go:64] FLAG: --register-node="true"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471501    2561 flags.go:64] FLAG: --register-schedulable="true"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471506    2561 flags.go:64] FLAG: --register-with-taints=""
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471512    2561 flags.go:64] FLAG: --registry-burst="10"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471517    2561 flags.go:64] FLAG: --registry-qps="5"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471522    2561 flags.go:64] FLAG: --reserved-cpus=""
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471527    2561 flags.go:64] FLAG: --reserved-memory=""
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471533    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471539    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471545    2561 flags.go:64] FLAG: --rotate-certificates="false"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471549    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471554    2561 flags.go:64] FLAG: --runonce="false"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471559    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471565    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471570    2561 flags.go:64] FLAG: --seccomp-default="false"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471575    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471582    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471587    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 26 16:00:53.478071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471592    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471597    2561 flags.go:64] FLAG: --storage-driver-password="root"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471602    2561 flags.go:64] FLAG: --storage-driver-secure="false"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471607    2561 flags.go:64] FLAG: --storage-driver-table="stats"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471611    2561 flags.go:64] FLAG: --storage-driver-user="root"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471616    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471621    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471626    2561 flags.go:64] FLAG: --system-cgroups=""
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471631    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471640    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471645    2561 flags.go:64] FLAG: --tls-cert-file=""
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471649    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471659    2561 flags.go:64] FLAG: --tls-min-version=""
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471664    2561 flags.go:64] FLAG: --tls-private-key-file=""
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471669    2561 flags.go:64] FLAG: --topology-manager-policy="none"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471673    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471678    2561 flags.go:64] FLAG: --topology-manager-scope="container"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471683    2561 flags.go:64] FLAG: --v="2"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471690    2561 flags.go:64] FLAG: --version="false"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471698    2561 flags.go:64] FLAG: --vmodule=""
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471704    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.471709    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471857    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471863    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:53.478803 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471868    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471872    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471876    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471880    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471884    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471903    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471910    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471914    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471918    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471923    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471928    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471932    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471936    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471940    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471945    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471949    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471953    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471957    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471961    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471966    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:53.479724 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471970    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471974    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471978    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471982    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471986    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471989    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471993    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.471997    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472003    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472008    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472014    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472019    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472023    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472027    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472031    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472035    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472039    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472044    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472050    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472055    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:53.480252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472060    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472064    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472069    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472073    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472077    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472081    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472085    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472089    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472093    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472097    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472104    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472110    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472114    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472119    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472123    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472127    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472134    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472140    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472145    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:53.480753 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472150    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472154    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472161    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472167    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472172    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472176    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472181    2561 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472186    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472191    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472195    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472199    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472205    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472209    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472213    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472217    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472221    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472226    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472230    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472235    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472239    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:53.481251 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472243    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472247    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472251    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472254    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.472259    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.472273    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.480494    2561 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.480512    2561 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480560    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480566    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480569    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480572    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480575    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480579    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480582    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:53.481729 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480585    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480588    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480591    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480593    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480596    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480598    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480601    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480604    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480607    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480609    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480612    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480615    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480618    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480622    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480626    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480629    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480633    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480635    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480639    2561 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:53.482180 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480641    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480644    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480647    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480650    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480653    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480657    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480659    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480662    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480665    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480667    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480670    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480672    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480675    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480678    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480681    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480683    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480686    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480689    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480691    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480694    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:53.482636 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480696    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480699    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480701    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480704    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480706    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480709    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480711    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480714    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480716    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480719    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480722    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480724    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480727    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480729    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480731    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480734    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480738    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480741    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480743    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480746    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:53.483248 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480748    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480751    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480754    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480756    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480759    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480762    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480765    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480768    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480771    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480775    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480779    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480782    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480785    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480788    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480790    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480793    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480795    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480798    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480800    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:53.483746 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480803    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.480808    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480927    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480933    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480936    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480939    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480942    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480945    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480948    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480951    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480954    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480957    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480959    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480962    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480965    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 26 16:00:53.484236 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480967    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480970    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480972    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480975    2561 feature_gate.go:328] unrecognized feature gate: Example2
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480978    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480980    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480983    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480985    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480989    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480992    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480995    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.480998    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481000    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481003    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481005    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481008    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481011    2561 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481013    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481016    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481018    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 26 16:00:53.484621 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481021    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481024    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481026    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481029    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481031    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481034    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481037    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481040    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481043    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481047    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481050    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481053    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481056    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481060    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481063    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481065    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481067    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481070    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481072    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481075    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 26 16:00:53.485192 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481077    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481080    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481083    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481086    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481089    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481092    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481094    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481096    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481099    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481101    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481104    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481106    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481109    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481111    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481114    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481117    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481119    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481122    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481125    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 26 16:00:53.485661 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481129    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481132    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481135    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481137    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481140    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481142    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481145    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481148    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481150    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481153    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481155    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481158    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481160    2561 feature_gate.go:328] unrecognized feature gate: Example
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:53.481163    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.481168    2561 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 26 16:00:53.486129 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.481935    2561 server.go:962] "Client rotation is on, will bootstrap in background"
May 26 16:00:53.486529 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.484455    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 26 16:00:53.486529 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.485253    2561 server.go:1019] "Starting client certificate rotation"
May 26 16:00:53.486529 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.485347    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:00:53.486529 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.485379    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 26 16:00:53.511628 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.511609    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:00:53.513514 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.513496    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 26 16:00:53.526442 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.526426    2561 log.go:25] "Validated CRI v1 runtime API"
May 26 16:00:53.531725 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.531709    2561 log.go:25] "Validated CRI v1 image API"
May 26 16:00:53.533007 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.532992    2561 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 26 16:00:53.537461 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.537307    2561 fs.go:135] Filesystem UUIDs: map[06c2c4e7-22c3-4931-9126-6e7732bc4c2f:/dev/nvme0n1p3 20369f7f-12cb-40dc-b741-8e9546b09283:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
May 26 16:00:53.537461 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.537380    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 26 16:00:53.542205 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.542101    2561 manager.go:217] Machine: {Timestamp:2026-05-26 16:00:53.541590019 +0000 UTC m=+0.413848444 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099597 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec211c104703af04aa075b876945b25f SystemUUID:ec211c10-4703-af04-aa07-5b876945b25f BootID:e2f28969-3895-4887-9d6d-4ba8a1f86c2e Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ad:62:c5:4b:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ad:62:c5:4b:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:3a:66:93:05:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 26 16:00:53.542825 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.542814    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 26 16:00:53.542964 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.542951    2561 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 26 16:00:53.545234 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.545207    2561 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 26 16:00:53.545378 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.545235    2561 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-131.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 26 16:00:53.545427 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.545384    2561 topology_manager.go:138] "Creating topology manager with none policy"
May 26 16:00:53.545427 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.545394    2561 container_manager_linux.go:306] "Creating device plugin manager"
May 26 16:00:53.545427 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.545401    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:00:53.546415 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.546394    2561 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 26 16:00:53.547755 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.547741    2561 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:00:53.547872 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.547863    2561 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 26 16:00:53.550308 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.550295    2561 kubelet.go:491] "Attempting to sync node with API server"
May 26 16:00:53.550349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.550318    2561 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 26 16:00:53.550349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.550331    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 26 16:00:53.550349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.550340    2561 kubelet.go:397] "Adding apiserver pod source"
May 26 16:00:53.550454 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.550356    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 26 16:00:53.551772 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.551759    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:00:53.551829 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.551786    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 26 16:00:53.552364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.552343    2561 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:00:53.555753 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.555739    2561 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 26 16:00:53.557704 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.557687    2561 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 26 16:00:53.558867 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558856    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 26 16:00:53.558929 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558875    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 26 16:00:53.558929 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558899    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 26 16:00:53.558929 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558906    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 26 16:00:53.558929 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558914    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 26 16:00:53.558929 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558920    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 26 16:00:53.558929 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558926    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 26 16:00:53.558929 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558931    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 26 16:00:53.559137 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558938    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 26 16:00:53.559137 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558945    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 26 16:00:53.559137 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.558959    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 26 16:00:53.559328 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.559314    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 26 16:00:53.560171 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.560161    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 26 16:00:53.560171 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.560172    2561 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 26 16:00:53.561849 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.561829    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 26 16:00:53.562044 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.562021    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 26 16:00:53.563357 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.563344    2561 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 26 16:00:53.563409 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.563383    2561 server.go:1295] "Started kubelet"
May 26 16:00:53.563502 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.563477    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 26 16:00:53.563555 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.563486    2561 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 26 16:00:53.563597 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.563571    2561 server_v1.go:47] "podresources" method="list" useActivePods=true
May 26 16:00:53.564295 ip-10-0-133-131 systemd[1]: Started Kubernetes Kubelet.
May 26 16:00:53.564781 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.564675    2561 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 26 16:00:53.566265 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.566249    2561 server.go:317] "Adding debug handlers to kubelet server"
May 26 16:00:53.571834 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.571787    2561 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-131.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 26 16:00:53.572089 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.572050    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 26 16:00:53.572294 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.572276    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 26 16:00:53.572605 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.572591    2561 volume_manager.go:295] "The desired_state_of_world populator starts"
May 26 16:00:53.572692 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.572683    2561 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 26 16:00:53.573091 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.572929    2561 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 26 16:00:53.573244 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.573225    2561 reconstruct.go:97] "Volume reconstruction finished"
May 26 16:00:53.573244 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.573247    2561 reconciler.go:26] "Reconciler: start to sync state"
May 26 16:00:53.573552 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.571789    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-131.ec2.internal.18b328e7bfe2fb12  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-131.ec2.internal,UID:ip-10-0-133-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-131.ec2.internal,},FirstTimestamp:2026-05-26 16:00:53.56335797 +0000 UTC m=+0.435616395,LastTimestamp:2026-05-26 16:00:53.56335797 +0000 UTC m=+0.435616395,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-131.ec2.internal,}"
May 26 16:00:53.573682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.573662    2561 factory.go:55] Registering systemd factory
May 26 16:00:53.573733 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.573686    2561 factory.go:223] Registration of the systemd container factory successfully
May 26 16:00:53.574192 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.574172    2561 factory.go:153] Registering CRI-O factory
May 26 16:00:53.574192 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.574192    2561 factory.go:223] Registration of the crio container factory successfully
May 26 16:00:53.574322 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.574286    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 26 16:00:53.574322 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.574307    2561 factory.go:103] Registering Raw factory
May 26 16:00:53.574382 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.574327    2561 manager.go:1196] Started watching for new ooms in manager
May 26 16:00:53.574495 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.574476    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:53.574913 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.574868    2561 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 26 16:00:53.575208 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.575193    2561 manager.go:319] Starting recovery of all containers
May 26 16:00:53.576862 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.575995    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4pcqp"
May 26 16:00:53.579028 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.579003    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-131.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
May 26 16:00:53.579301 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.579275    2561 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
May 26 16:00:53.580771 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.580735    2561 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 26 16:00:53.586394 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.586292    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4pcqp"
May 26 16:00:53.587526 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.587455    2561 manager.go:324] Recovery completed
May 26 16:00:53.591424 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.591411    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:53.593373 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.593359    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:53.593432 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.593384    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:53.593432 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.593394    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:53.593875 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.593859    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 26 16:00:53.593875 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.593873    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 26 16:00:53.594000 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.593907    2561 state_mem.go:36] "Initialized new in-memory state store"
May 26 16:00:53.595284 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.595224    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-131.ec2.internal.18b328e7c1acf704  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-131.ec2.internal,UID:ip-10-0-133-131.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-133-131.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-133-131.ec2.internal,},FirstTimestamp:2026-05-26 16:00:53.59337242 +0000 UTC m=+0.465630844,LastTimestamp:2026-05-26 16:00:53.59337242 +0000 UTC m=+0.465630844,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-131.ec2.internal,}"
May 26 16:00:53.596197 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.596182    2561 policy_none.go:49] "None policy: Start"
May 26 16:00:53.596197 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.596200    2561 memory_manager.go:186] "Starting memorymanager" policy="None"
May 26 16:00:53.596285 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.596210    2561 state_mem.go:35] "Initializing new in-memory state store"
May 26 16:00:53.632510 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.632495    2561 manager.go:341] "Starting Device Plugin manager"
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.632533    2561 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.632544    2561 server.go:85] "Starting device plugin registration server"
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.632818    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.632831    2561 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.632933    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.633078    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.633087    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.633448    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 26 16:00:53.647286 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.633485    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:53.706801 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.706763    2561 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 26 16:00:53.706801 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.706793    2561 status_manager.go:230] "Starting to sync pod status with apiserver"
May 26 16:00:53.706980 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.706825    2561 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 26 16:00:53.706980 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.706833    2561 kubelet.go:2451] "Starting kubelet main sync loop"
May 26 16:00:53.706980 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.706870    2561 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 26 16:00:53.709534 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.709517    2561 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:53.733642 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.733585    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:53.734747 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.734730    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:53.734842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.734758    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:53.734842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.734769    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:53.734842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.734801    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-131.ec2.internal"
May 26 16:00:53.746325 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.746311    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-131.ec2.internal"
May 26 16:00:53.746368 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.746333    2561 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-131.ec2.internal\": node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:53.778266 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.778246    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:53.807711 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.807687    2561 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"]
May 26 16:00:53.807782 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.807750    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:53.809827 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.809813    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:53.809921 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.809838    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:53.809921 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.809848    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:53.811417 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.811405    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:53.811552 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.811538    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.811587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.811569    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:53.812819 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.812798    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:53.812907 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.812833    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:53.812907 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.812846    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:53.812973 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.812807    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:53.812973 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.812936    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:53.812973 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.812951    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:53.814250 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.814232    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.814337 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.814258    2561 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 26 16:00:53.816481 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.816466    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientMemory"
May 26 16:00:53.816557 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.816494    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasNoDiskPressure"
May 26 16:00:53.816557 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.816506    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeHasSufficientPID"
May 26 16:00:53.833426 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.833405    2561 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-131.ec2.internal\" not found" node="ip-10-0-133-131.ec2.internal"
May 26 16:00:53.836857 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.836843    2561 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-131.ec2.internal\" not found" node="ip-10-0-133-131.ec2.internal"
May 26 16:00:53.874655 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.874614    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7eb90e3821d9b3bc90d7a8d213d3b8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"a7eb90e3821d9b3bc90d7a8d213d3b8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.874655 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.874652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7eb90e3821d9b3bc90d7a8d213d3b8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"a7eb90e3821d9b3bc90d7a8d213d3b8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.874851 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.874668    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e74c204e9201dd85fbf7a07770c47c82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"e74c204e9201dd85fbf7a07770c47c82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.879214 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.879195    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:53.974995 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.974953    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7eb90e3821d9b3bc90d7a8d213d3b8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"a7eb90e3821d9b3bc90d7a8d213d3b8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.974995 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.974994    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e74c204e9201dd85fbf7a07770c47c82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"e74c204e9201dd85fbf7a07770c47c82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.975211 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.975018    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7eb90e3821d9b3bc90d7a8d213d3b8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"a7eb90e3821d9b3bc90d7a8d213d3b8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.975211 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.975062    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a7eb90e3821d9b3bc90d7a8d213d3b8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"a7eb90e3821d9b3bc90d7a8d213d3b8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.975211 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.975068    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7eb90e3821d9b3bc90d7a8d213d3b8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal\" (UID: \"a7eb90e3821d9b3bc90d7a8d213d3b8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.975211 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:53.975068    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e74c204e9201dd85fbf7a07770c47c82-config\") pod \"kube-apiserver-proxy-ip-10-0-133-131.ec2.internal\" (UID: \"e74c204e9201dd85fbf7a07770c47c82\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
May 26 16:00:53.980031 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:53.980015    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.080850 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:54.080809    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.137011 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.136983    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:54.139864 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.139847    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
May 26 16:00:54.181949 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:54.181910    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.282498 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:54.282458    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.383168 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:54.383076    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.483707 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:54.483677    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.485822 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.485804    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 26 16:00:54.485993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.485976    2561 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 26 16:00:54.572440 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.572416    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 26 16:00:54.583835 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:54.583808    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.588634 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.588576    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-25 15:55:53 +0000 UTC" deadline="2028-01-17 18:15:09.797748327 +0000 UTC"
May 26 16:00:54.588634 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.588611    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14426h14m15.209141888s"
May 26 16:00:54.591977 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.591957    2561 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 26 16:00:54.642432 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.642370    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x74sg"
May 26 16:00:54.659812 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.659778    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x74sg"
May 26 16:00:54.684497 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:54.684465    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.707982 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:54.707943    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74c204e9201dd85fbf7a07770c47c82.slice/crio-1b44db25c8b2277e3dd73b09e827f57fa49b3866e80028c03e3da7202318a588 WatchSource:0}: Error finding container 1b44db25c8b2277e3dd73b09e827f57fa49b3866e80028c03e3da7202318a588: Status 404 returned error can't find the container with id 1b44db25c8b2277e3dd73b09e827f57fa49b3866e80028c03e3da7202318a588
May 26 16:00:54.711695 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.711604    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"a7eb90e3821d9b3bc90d7a8d213d3b8f","Type":"ContainerStarted","Data":"6ce7311d290e3004d0cb1b2a6edd0732775ce4ce67ceec1b913ac4cc1cf198ad"}
May 26 16:00:54.711968 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.711953    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:00:54.785303 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:54.785253    2561 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-133-131.ec2.internal\" not found"
May 26 16:00:54.878422 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.878393    2561 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:54.974733 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.974646    2561 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal"
May 26 16:00:54.986556 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.986524    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:00:54.988769 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:54.988745    2561 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal"
May 26 16:00:55.000031 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.000005    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 26 16:00:55.047442 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.047411    2561 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:55.171393 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.171362    2561 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:55.551400 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.551359    2561 apiserver.go:52] "Watching apiserver"
May 26 16:00:55.560680 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.560654    2561 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 26 16:00:55.563143 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.563109    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-swnch","openshift-network-operator/iptables-alerter-scmxr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl","openshift-cluster-node-tuning-operator/tuned-jhz6k","openshift-dns/node-resolver-52kjn","openshift-multus/multus-fj8z4","openshift-ovn-kubernetes/ovnkube-node-jslmk","kube-system/konnectivity-agent-c8tfr","kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal","openshift-image-registry/node-ca-wj5zg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal","openshift-multus/multus-additional-cni-plugins-vn5k7","openshift-multus/network-metrics-daemon-6glcl"]
May 26 16:00:55.566079 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.566057    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.568453 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.568431    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.570209 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.569738    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 26 16:00:55.570209 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.569865    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rgtn7\""
May 26 16:00:55.570209 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.569987    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 26 16:00:55.570209 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.570159    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 26 16:00:55.570464 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.570265    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 26 16:00:55.570464 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.570297    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 26 16:00:55.570464 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.570344    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 26 16:00:55.572318 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.570789    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 26 16:00:55.572318 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.571282    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vczzv\""
May 26 16:00:55.572318 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.571309    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 26 16:00:55.572318 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.571567    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 26 16:00:55.572703 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.572624    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.574964 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.574943    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 26 16:00:55.575069 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.575057    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 26 16:00:55.575184 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.575168    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 26 16:00:55.575300 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.575219    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gmxkn\""
May 26 16:00:55.577346 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.577326    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.580372 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.580339    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 26 16:00:55.580372 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.580366    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 26 16:00:55.580576 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.580556    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p7mrf\""
May 26 16:00:55.581199 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.581167    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.583140 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583116    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-ovn\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.583230 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583160    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.583230 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovnkube-script-lib\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.583230 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-kubernetes\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.583390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-slash\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.583390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.583390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-device-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.583390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-systemd\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.583390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583339    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 26 16:00:55.583390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583345    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 26 16:00:55.583390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-lib-modules\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583392    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-run-netns\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583346    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9c79z\""
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-etc-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583452    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583507    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-cni-netd\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554d928f-b65a-4f7f-ace1-b4e49f30093c-host-slash\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583564    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583589    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-sys\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-var-lib-kubelet\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583637    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-tmp\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583662    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jvjx\" (UniqueName: \"kubernetes.io/projected/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-kube-api-access-7jvjx\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.583710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583684    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-kubelet\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583718    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-node-log\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldkml\" (UniqueName: \"kubernetes.io/projected/554d928f-b65a-4f7f-ace1-b4e49f30093c-kube-api-access-ldkml\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583760    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-tuned\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583780    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-var-lib-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/554d928f-b65a-4f7f-ace1-b4e49f30093c-iptables-alerter-script\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583839    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysctl-d\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583921    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-host\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.583976    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-systemd-units\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584005    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-log-socket\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-run-ovn-kubernetes\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584081    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-env-overrides\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584100    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pr64\" (UniqueName: \"kubernetes.io/projected/56b37ce7-4b56-469f-a8a7-c55247d53a12-kube-api-access-2pr64\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584126    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysctl-conf\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584187    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-run\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-systemd\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-cni-bin\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584241    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovnkube-config\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovn-node-metrics-cert\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.584986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-modprobe-d\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.584986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584340    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysconfig\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.584986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584366    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znmfm\" (UniqueName: \"kubernetes.io/projected/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-kube-api-access-znmfm\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.584986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584412    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.584986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.584436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.585671 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.585651    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.585788 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.585770    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:00:55.585921 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:55.585876    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:00:55.587957 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.587936    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bnzht\""
May 26 16:00:55.588215 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.588198    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 26 16:00:55.588296 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.588242    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 26 16:00:55.588365 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.588200    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 26 16:00:55.588554 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.588534    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 26 16:00:55.590554 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.590531    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:00:55.592949 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.592926    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.593049 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.593020    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.595593 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.595454    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:55.595593 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:55.595521    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:00:55.595593 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.595552    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gd4dh\""
May 26 16:00:55.595804 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.595599    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fdt45\""
May 26 16:00:55.595804 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.595746    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 26 16:00:55.595920 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.595815    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 26 16:00:55.596024 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.596007    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 26 16:00:55.596024 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.596021    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 26 16:00:55.596261 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.596242    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 26 16:00:55.597001 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.596832    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 26 16:00:55.597001 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.596901    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 26 16:00:55.597170 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.597151    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jczks\""
May 26 16:00:55.660497 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.660468    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:54 +0000 UTC" deadline="2028-01-01 02:49:30.260948551 +0000 UTC"
May 26 16:00:55.660497 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.660494    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14026h48m34.600456722s"
May 26 16:00:55.674399 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.674374    2561 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 26 16:00:55.685473 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685437    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f685ebf-142e-4075-888a-d44a456c5812-host\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.685583 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f685ebf-142e-4075-888a-d44a456c5812-serviceca\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.685583 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.685583 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685540    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysctl-d\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.685583 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685556    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-host\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.685583 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685574    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-systemd-units\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685591    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2pr64\" (UniqueName: \"kubernetes.io/projected/56b37ce7-4b56-469f-a8a7-c55247d53a12-kube-api-access-2pr64\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685638    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-cnibin\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685654    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685650    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-host\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685680    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-run\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685679    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-systemd-units\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685731    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-systemd\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.685821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685699    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysctl-d\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685854    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-cni-bin\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685876    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-run\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685901    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovnkube-config\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685929    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-systemd\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.685945    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-cni-bin\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-cni-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686087    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-system-cni-dir\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686117    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-ovn\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wb6x\" (UniqueName: \"kubernetes.io/projected/37120c41-44fb-484a-b6e6-f6fe6e962658-kube-api-access-7wb6x\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686163    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-slash\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686197    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.686241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686204    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-ovn\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686257    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-systemd\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686268    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-slash\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686310    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-lib-modules\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686324    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-socket-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686338    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-run-netns\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686369    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-etc-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686371    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-run-netns\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686394    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686313    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-systemd\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686400    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-etc-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686438    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686441    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-lib-modules\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-cnibin\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686459    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-run-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-netns\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686503    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-registration-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.686794 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686501    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovnkube-config\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686524    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-tmp\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-conf-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686596    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-os-release\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686624    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686667    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-var-lib-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686706    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61d768a5-d28e-4ac8-8d64-9a8ab4718e68-agent-certs\") pod \"konnectivity-agent-c8tfr\" (UID: \"61d768a5-d28e-4ac8-8d64-9a8ab4718e68\") " pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686738    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-var-lib-openvswitch\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61d768a5-d28e-4ac8-8d64-9a8ab4718e68-konnectivity-ca\") pod \"konnectivity-agent-c8tfr\" (UID: \"61d768a5-d28e-4ac8-8d64-9a8ab4718e68\") " pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686805    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37120c41-44fb-484a-b6e6-f6fe6e962658-tmp-dir\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686807    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686835    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-log-socket\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686860    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-run-ovn-kubernetes\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686902    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-env-overrides\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686911    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-log-socket\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.686973    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-run-ovn-kubernetes\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687011    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-k8s-cni-cncf-io\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.687475 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687039    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7vt\" (UniqueName: \"kubernetes.io/projected/33633aed-0f35-47bb-9be8-158d6d96a114-kube-api-access-jq7vt\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687063    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbqxs\" (UniqueName: \"kubernetes.io/projected/e29c2933-d508-4230-9a48-47e2758672d5-kube-api-access-sbqxs\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687090    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysctl-conf\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687147    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovn-node-metrics-cert\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687216    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687244    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687255    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-env-overrides\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687328    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-sys-fs\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-multus-certs\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687336    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-etc-selinux\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687327    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysctl-conf\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687365    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwrt8\" (UniqueName: \"kubernetes.io/projected/3f685ebf-142e-4075-888a-d44a456c5812-kube-api-access-wwrt8\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687396    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj98m\" (UniqueName: \"kubernetes.io/projected/4fbe1d25-e242-4838-b10c-78c4a9c1196e-kube-api-access-tj98m\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687429    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-modprobe-d\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687455    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysconfig\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687481    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-znmfm\" (UniqueName: \"kubernetes.io/projected/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-kube-api-access-znmfm\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687504    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.688246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-cni-multus\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687551    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-sysconfig\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687560    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687592    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovnkube-script-lib\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687614    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-os-release\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687635    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687648    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33633aed-0f35-47bb-9be8-158d6d96a114-cni-binary-copy\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687685    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-socket-dir-parent\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687593    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-modprobe-d\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687745    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-kubernetes\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687770    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-device-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687799    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33633aed-0f35-47bb-9be8-158d6d96a114-multus-daemon-config\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687816    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-kubernetes\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687822    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-cni-netd\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687848    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554d928f-b65a-4f7f-ace1-b4e49f30093c-host-slash\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.689061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-kubelet\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687900    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/56b37ce7-4b56-469f-a8a7-c55247d53a12-device-dir\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687908    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-cni-netd\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687912    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-etc-kubernetes\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687955    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37120c41-44fb-484a-b6e6-f6fe6e962658-hosts-file\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687962    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/554d928f-b65a-4f7f-ace1-b4e49f30093c-host-slash\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.687987    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-sys\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688027    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-var-lib-kubelet\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688037    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-sys\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688055    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7jvjx\" (UniqueName: \"kubernetes.io/projected/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-kube-api-access-7jvjx\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688081    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-kubelet\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688108    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-node-log\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688123    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-host-kubelet\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688136    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ldkml\" (UniqueName: \"kubernetes.io/projected/554d928f-b65a-4f7f-ace1-b4e49f30093c-kube-api-access-ldkml\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688243    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-hostroot\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688083    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-var-lib-kubelet\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688176    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-node-log\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.689806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688281    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688285    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovnkube-script-lib\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688357    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-tuned\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688389    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/554d928f-b65a-4f7f-ace1-b4e49f30093c-iptables-alerter-script\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-system-cni-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688463    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-cni-bin\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.688863    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/554d928f-b65a-4f7f-ace1-b4e49f30093c-iptables-alerter-script\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.690229    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-tmp\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.690390    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-ovn-node-metrics-cert\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.690532 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.690501    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-etc-tuned\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.698651 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.698528    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pr64\" (UniqueName: \"kubernetes.io/projected/56b37ce7-4b56-469f-a8a7-c55247d53a12-kube-api-access-2pr64\") pod \"aws-ebs-csi-driver-node-6q9bl\" (UID: \"56b37ce7-4b56-469f-a8a7-c55247d53a12\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.698651 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.698570    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jvjx\" (UniqueName: \"kubernetes.io/projected/c096ec5a-91c6-4ddb-ad74-6d92cf1396f9-kube-api-access-7jvjx\") pod \"tuned-jhz6k\" (UID: \"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9\") " pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.698810 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.698759    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldkml\" (UniqueName: \"kubernetes.io/projected/554d928f-b65a-4f7f-ace1-b4e49f30093c-kube-api-access-ldkml\") pod \"iptables-alerter-scmxr\" (UID: \"554d928f-b65a-4f7f-ace1-b4e49f30093c\") " pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.698848 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.698828    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znmfm\" (UniqueName: \"kubernetes.io/projected/f9ddde5e-182c-4cd4-b95e-f80ef8fdac39-kube-api-access-znmfm\") pod \"ovnkube-node-jslmk\" (UID: \"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39\") " pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.714038 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.714015    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" event={"ID":"e74c204e9201dd85fbf7a07770c47c82","Type":"ContainerStarted","Data":"1b44db25c8b2277e3dd73b09e827f57fa49b3866e80028c03e3da7202318a588"}
May 26 16:00:55.788938 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.788877    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789120 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.788946    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-system-cni-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.789120 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.788976    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-cni-bin\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.789120 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.788999    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f685ebf-142e-4075-888a-d44a456c5812-host\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.789120 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789023    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f685ebf-142e-4075-888a-d44a456c5812-serviceca\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.789120 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789047    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789120 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789075    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:00:55.789120 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789090    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-system-cni-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.789120 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789100    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-cnibin\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789139    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-cnibin\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789156    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789186    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-cni-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789206    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-system-cni-dir\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789217    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3f685ebf-142e-4075-888a-d44a456c5812-host\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789228    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7wb6x\" (UniqueName: \"kubernetes.io/projected/37120c41-44fb-484a-b6e6-f6fe6e962658-kube-api-access-7wb6x\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789263    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-cnibin\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789288    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-netns\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789309    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-conf-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789331    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-os-release\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789354    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789387    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61d768a5-d28e-4ac8-8d64-9a8ab4718e68-agent-certs\") pod \"konnectivity-agent-c8tfr\" (UID: \"61d768a5-d28e-4ac8-8d64-9a8ab4718e68\") " pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789413    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61d768a5-d28e-4ac8-8d64-9a8ab4718e68-konnectivity-ca\") pod \"konnectivity-agent-c8tfr\" (UID: \"61d768a5-d28e-4ac8-8d64-9a8ab4718e68\") " pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789441    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37120c41-44fb-484a-b6e6-f6fe6e962658-tmp-dir\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789472    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-k8s-cni-cncf-io\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789478    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-cni-binary-copy\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.789498 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789507    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7vt\" (UniqueName: \"kubernetes.io/projected/33633aed-0f35-47bb-9be8-158d6d96a114-kube-api-access-jq7vt\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789538    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sbqxs\" (UniqueName: \"kubernetes.io/projected/e29c2933-d508-4230-9a48-47e2758672d5-kube-api-access-sbqxs\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789570    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-multus-certs\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789588    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wwrt8\" (UniqueName: \"kubernetes.io/projected/3f685ebf-142e-4075-888a-d44a456c5812-kube-api-access-wwrt8\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789607    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tj98m\" (UniqueName: \"kubernetes.io/projected/4fbe1d25-e242-4838-b10c-78c4a9c1196e-kube-api-access-tj98m\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789627    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3f685ebf-142e-4075-888a-d44a456c5812-serviceca\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789630    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-cni-multus\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789682    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-os-release\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789711    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33633aed-0f35-47bb-9be8-158d6d96a114-cni-binary-copy\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789723    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-conf-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789736    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-socket-dir-parent\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789766    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789785    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-os-release\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789794    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33633aed-0f35-47bb-9be8-158d6d96a114-multus-daemon-config\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789823    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-kubelet\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789850    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-etc-kubernetes\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789874    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37120c41-44fb-484a-b6e6-f6fe6e962658-hosts-file\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789925    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-hostroot\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.790283 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789957    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790019    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-hostroot\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790079    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-cni-dir\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790084    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789186    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-cni-bin\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790129    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e29c2933-d508-4230-9a48-47e2758672d5-system-cni-dir\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790259    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e29c2933-d508-4230-9a48-47e2758672d5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790311    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-cnibin\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:55.790397    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:55.790474    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs podName:4fbe1d25-e242-4838-b10c-78c4a9c1196e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:56.290453977 +0000 UTC m=+3.162712412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs") pod "network-metrics-daemon-6glcl" (UID: "4fbe1d25-e242-4838-b10c-78c4a9c1196e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790535    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-os-release\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790585    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-multus-socket-dir-parent\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.789684    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-cni-multus\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790632    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-netns\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790662    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-multus-certs\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790683    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-etc-kubernetes\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790699    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-var-lib-kubelet\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790763    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/37120c41-44fb-484a-b6e6-f6fe6e962658-hosts-file\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.791021 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.790772    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33633aed-0f35-47bb-9be8-158d6d96a114-host-run-k8s-cni-cncf-io\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791651 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.791074    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37120c41-44fb-484a-b6e6-f6fe6e962658-tmp-dir\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.791651 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.791213    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/61d768a5-d28e-4ac8-8d64-9a8ab4718e68-konnectivity-ca\") pod \"konnectivity-agent-c8tfr\" (UID: \"61d768a5-d28e-4ac8-8d64-9a8ab4718e68\") " pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:00:55.791651 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.791294    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33633aed-0f35-47bb-9be8-158d6d96a114-multus-daemon-config\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.791651 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.791426    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33633aed-0f35-47bb-9be8-158d6d96a114-cni-binary-copy\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.793589 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.793569    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/61d768a5-d28e-4ac8-8d64-9a8ab4718e68-agent-certs\") pod \"konnectivity-agent-c8tfr\" (UID: \"61d768a5-d28e-4ac8-8d64-9a8ab4718e68\") " pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:00:55.797066 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:55.797035    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:55.797066 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:55.797057    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:55.797066 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:55.797071    2561 projected.go:194] Error preparing data for projected volume kube-api-access-4fsct for pod openshift-network-diagnostics/network-check-target-swnch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:55.797297 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:55.797132    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct podName:58051681-235b-4f37-ae82-7b6105f0c847 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:56.297114381 +0000 UTC m=+3.169372830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4fsct" (UniqueName: "kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct") pod "network-check-target-swnch" (UID: "58051681-235b-4f37-ae82-7b6105f0c847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:55.798483 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.798461    2561 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 26 16:00:55.799560 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.799536    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wb6x\" (UniqueName: \"kubernetes.io/projected/37120c41-44fb-484a-b6e6-f6fe6e962658-kube-api-access-7wb6x\") pod \"node-resolver-52kjn\" (UID: \"37120c41-44fb-484a-b6e6-f6fe6e962658\") " pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.799752 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.799731    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7vt\" (UniqueName: \"kubernetes.io/projected/33633aed-0f35-47bb-9be8-158d6d96a114-kube-api-access-jq7vt\") pod \"multus-fj8z4\" (UID: \"33633aed-0f35-47bb-9be8-158d6d96a114\") " pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.799878 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.799852    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj98m\" (UniqueName: \"kubernetes.io/projected/4fbe1d25-e242-4838-b10c-78c4a9c1196e-kube-api-access-tj98m\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:55.800721 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.800698    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwrt8\" (UniqueName: \"kubernetes.io/projected/3f685ebf-142e-4075-888a-d44a456c5812-kube-api-access-wwrt8\") pod \"node-ca-wj5zg\" (UID: \"3f685ebf-142e-4075-888a-d44a456c5812\") " pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.800855 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.800834    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbqxs\" (UniqueName: \"kubernetes.io/projected/e29c2933-d508-4230-9a48-47e2758672d5-kube-api-access-sbqxs\") pod \"multus-additional-cni-plugins-vn5k7\" (UID: \"e29c2933-d508-4230-9a48-47e2758672d5\") " pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:55.880981 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.880880    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:00:55.889657 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.889633    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-scmxr"
May 26 16:00:55.900315 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.900291    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl"
May 26 16:00:55.906255 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.906231    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhz6k"
May 26 16:00:55.917812 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.917790    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-52kjn"
May 26 16:00:55.925416 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.925397    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fj8z4"
May 26 16:00:55.932985 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.932965    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:00:55.938507 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.938491    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wj5zg"
May 26 16:00:55.944098 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:55.944076    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vn5k7"
May 26 16:00:56.256138 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.255919    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ddde5e_182c_4cd4_b95e_f80ef8fdac39.slice/crio-9ade9dfa5c650216008d784ce74cf9f8058c8600cacdd8967db955131cbbbc98 WatchSource:0}: Error finding container 9ade9dfa5c650216008d784ce74cf9f8058c8600cacdd8967db955131cbbbc98: Status 404 returned error can't find the container with id 9ade9dfa5c650216008d784ce74cf9f8058c8600cacdd8967db955131cbbbc98
May 26 16:00:56.257242 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.256766    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29c2933_d508_4230_9a48_47e2758672d5.slice/crio-fb399311cd1bb6073c753ec121caa21ec1e743f5777f9a411222c789e9d75dcb WatchSource:0}: Error finding container fb399311cd1bb6073c753ec121caa21ec1e743f5777f9a411222c789e9d75dcb: Status 404 returned error can't find the container with id fb399311cd1bb6073c753ec121caa21ec1e743f5777f9a411222c789e9d75dcb
May 26 16:00:56.261144 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.261124    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37120c41_44fb_484a_b6e6_f6fe6e962658.slice/crio-6ef271f840ffcbbd15ff9a07a1a0d97a765204811b274ed2a04eb7a9134075fe WatchSource:0}: Error finding container 6ef271f840ffcbbd15ff9a07a1a0d97a765204811b274ed2a04eb7a9134075fe: Status 404 returned error can't find the container with id 6ef271f840ffcbbd15ff9a07a1a0d97a765204811b274ed2a04eb7a9134075fe
May 26 16:00:56.261914 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.261839    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b37ce7_4b56_469f_a8a7_c55247d53a12.slice/crio-b955dd62b56788d82fbe9d343f9d4d9f76382290f5f4f01aee6ec9c18531b78f WatchSource:0}: Error finding container b955dd62b56788d82fbe9d343f9d4d9f76382290f5f4f01aee6ec9c18531b78f: Status 404 returned error can't find the container with id b955dd62b56788d82fbe9d343f9d4d9f76382290f5f4f01aee6ec9c18531b78f
May 26 16:00:56.262775 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.262722    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f685ebf_142e_4075_888a_d44a456c5812.slice/crio-de8339654f31914acc26b4ee791efbd103a42605c260c7e94a70de1b8430df11 WatchSource:0}: Error finding container de8339654f31914acc26b4ee791efbd103a42605c260c7e94a70de1b8430df11: Status 404 returned error can't find the container with id de8339654f31914acc26b4ee791efbd103a42605c260c7e94a70de1b8430df11
May 26 16:00:56.263733 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.263710    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc096ec5a_91c6_4ddb_ad74_6d92cf1396f9.slice/crio-3703037c604e67df4d5bf9426e7d8a14e614acfc3a564b4bcfc066c8085258ff WatchSource:0}: Error finding container 3703037c604e67df4d5bf9426e7d8a14e614acfc3a564b4bcfc066c8085258ff: Status 404 returned error can't find the container with id 3703037c604e67df4d5bf9426e7d8a14e614acfc3a564b4bcfc066c8085258ff
May 26 16:00:56.264612 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.264586    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33633aed_0f35_47bb_9be8_158d6d96a114.slice/crio-294597273ce0a5d44205beaa2d31b08291906012ff2226815bfa32c12fd9c02d WatchSource:0}: Error finding container 294597273ce0a5d44205beaa2d31b08291906012ff2226815bfa32c12fd9c02d: Status 404 returned error can't find the container with id 294597273ce0a5d44205beaa2d31b08291906012ff2226815bfa32c12fd9c02d
May 26 16:00:56.265579 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.265487    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554d928f_b65a_4f7f_ace1_b4e49f30093c.slice/crio-7e8dbfb4d1a4d9ead5e6a16ae538a6fb22a3c69ec2a13f2a0a655cff85456fe3 WatchSource:0}: Error finding container 7e8dbfb4d1a4d9ead5e6a16ae538a6fb22a3c69ec2a13f2a0a655cff85456fe3: Status 404 returned error can't find the container with id 7e8dbfb4d1a4d9ead5e6a16ae538a6fb22a3c69ec2a13f2a0a655cff85456fe3
May 26 16:00:56.267023 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:00:56.266965    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d768a5_d28e_4ac8_8d64_9a8ab4718e68.slice/crio-b748cd8dfcea036ac3c05cac9a0c99acbb346aac69ccf9fc6cd4de8b9a8a9e40 WatchSource:0}: Error finding container b748cd8dfcea036ac3c05cac9a0c99acbb346aac69ccf9fc6cd4de8b9a8a9e40: Status 404 returned error can't find the container with id b748cd8dfcea036ac3c05cac9a0c99acbb346aac69ccf9fc6cd4de8b9a8a9e40
May 26 16:00:56.292565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.292541    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:56.292702 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:56.292683    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:56.292748 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:56.292743    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs podName:4fbe1d25-e242-4838-b10c-78c4a9c1196e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.292727865 +0000 UTC m=+4.164986277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs") pod "network-metrics-daemon-6glcl" (UID: "4fbe1d25-e242-4838-b10c-78c4a9c1196e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:56.393856 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.393818    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:00:56.394030 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:56.394010    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:56.394073 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:56.394035    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:56.394073 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:56.394045    2561 projected.go:194] Error preparing data for projected volume kube-api-access-4fsct for pod openshift-network-diagnostics/network-check-target-swnch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:56.394134 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:56.394114    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct podName:58051681-235b-4f37-ae82-7b6105f0c847 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:57.394087984 +0000 UTC m=+4.266346402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fsct" (UniqueName: "kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct") pod "network-check-target-swnch" (UID: "58051681-235b-4f37-ae82-7b6105f0c847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:56.661528 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.661474    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-25 15:55:54 +0000 UTC" deadline="2027-12-25 23:06:32.218365028 +0000 UTC"
May 26 16:00:56.661528 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.661515    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13879h5m35.55685396s"
May 26 16:00:56.723664 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.723587    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52kjn" event={"ID":"37120c41-44fb-484a-b6e6-f6fe6e962658","Type":"ContainerStarted","Data":"6ef271f840ffcbbd15ff9a07a1a0d97a765204811b274ed2a04eb7a9134075fe"}
May 26 16:00:56.727287 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.727223    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" event={"ID":"e29c2933-d508-4230-9a48-47e2758672d5","Type":"ContainerStarted","Data":"fb399311cd1bb6073c753ec121caa21ec1e743f5777f9a411222c789e9d75dcb"}
May 26 16:00:56.731934 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.731873    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"9ade9dfa5c650216008d784ce74cf9f8058c8600cacdd8967db955131cbbbc98"}
May 26 16:00:56.737877 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.737043    2561 generic.go:358] "Generic (PLEG): container finished" podID="a7eb90e3821d9b3bc90d7a8d213d3b8f" containerID="60e4ed7d6b4be6c056e0464f0fe92fef2563ee628552edc97fd232cf2b8919b6" exitCode=0
May 26 16:00:56.737877 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.737616    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"a7eb90e3821d9b3bc90d7a8d213d3b8f","Type":"ContainerDied","Data":"60e4ed7d6b4be6c056e0464f0fe92fef2563ee628552edc97fd232cf2b8919b6"}
May 26 16:00:56.742828 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.741206    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" event={"ID":"e74c204e9201dd85fbf7a07770c47c82","Type":"ContainerStarted","Data":"47e706dfdd4c6988b211d9525e7036a73feef7ec7722311ec6d922f2a33f4953"}
May 26 16:00:56.747209 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.747127    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c8tfr" event={"ID":"61d768a5-d28e-4ac8-8d64-9a8ab4718e68","Type":"ContainerStarted","Data":"b748cd8dfcea036ac3c05cac9a0c99acbb346aac69ccf9fc6cd4de8b9a8a9e40"}
May 26 16:00:56.752137 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.752054    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-scmxr" event={"ID":"554d928f-b65a-4f7f-ace1-b4e49f30093c","Type":"ContainerStarted","Data":"7e8dbfb4d1a4d9ead5e6a16ae538a6fb22a3c69ec2a13f2a0a655cff85456fe3"}
May 26 16:00:56.762085 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.762037    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhz6k" event={"ID":"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9","Type":"ContainerStarted","Data":"3703037c604e67df4d5bf9426e7d8a14e614acfc3a564b4bcfc066c8085258ff"}
May 26 16:00:56.768444 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.768397    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wj5zg" event={"ID":"3f685ebf-142e-4075-888a-d44a456c5812","Type":"ContainerStarted","Data":"de8339654f31914acc26b4ee791efbd103a42605c260c7e94a70de1b8430df11"}
May 26 16:00:56.772150 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.772087    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fj8z4" event={"ID":"33633aed-0f35-47bb-9be8-158d6d96a114","Type":"ContainerStarted","Data":"294597273ce0a5d44205beaa2d31b08291906012ff2226815bfa32c12fd9c02d"}
May 26 16:00:56.789255 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:56.789056    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl" event={"ID":"56b37ce7-4b56-469f-a8a7-c55247d53a12","Type":"ContainerStarted","Data":"b955dd62b56788d82fbe9d343f9d4d9f76382290f5f4f01aee6ec9c18531b78f"}
May 26 16:00:57.301072 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:57.301033    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:57.301262 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:57.301191    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:57.301333 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:57.301270    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs podName:4fbe1d25-e242-4838-b10c-78c4a9c1196e nodeName:}" failed. No retries permitted until 2026-05-26 16:00:59.301248261 +0000 UTC m=+6.173506687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs") pod "network-metrics-daemon-6glcl" (UID: "4fbe1d25-e242-4838-b10c-78c4a9c1196e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:57.402360 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:57.402309    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:00:57.402552 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:57.402487    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:57.402552 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:57.402508    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:57.402552 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:57.402521    2561 projected.go:194] Error preparing data for projected volume kube-api-access-4fsct for pod openshift-network-diagnostics/network-check-target-swnch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:57.402710 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:57.402578    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct podName:58051681-235b-4f37-ae82-7b6105f0c847 nodeName:}" failed. No retries permitted until 2026-05-26 16:00:59.402561135 +0000 UTC m=+6.274819551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fsct" (UniqueName: "kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct") pod "network-check-target-swnch" (UID: "58051681-235b-4f37-ae82-7b6105f0c847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:57.707986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:57.707825    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:00:57.709201 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:57.708586    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:00:57.709201 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:57.709027    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:57.709201 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:57.709151    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:00:57.795601 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:57.795564    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" event={"ID":"a7eb90e3821d9b3bc90d7a8d213d3b8f","Type":"ContainerStarted","Data":"949aa68f709a68d756d16c06b8f7d4a249e1cc8f7f1efa923cea3d410cb4f6b8"}
May 26 16:00:57.811616 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:57.811309    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-131.ec2.internal" podStartSLOduration=3.81128782 podStartE2EDuration="3.81128782s" podCreationTimestamp="2026-05-26 16:00:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:56.767279655 +0000 UTC m=+3.639538091" watchObservedRunningTime="2026-05-26 16:00:57.81128782 +0000 UTC m=+4.683546255"
May 26 16:00:59.317857 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:59.317285    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:59.317857 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:59.317436    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:59.317857 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:59.317502    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs podName:4fbe1d25-e242-4838-b10c-78c4a9c1196e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:03.317483282 +0000 UTC m=+10.189741717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs") pod "network-metrics-daemon-6glcl" (UID: "4fbe1d25-e242-4838-b10c-78c4a9c1196e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:00:59.418319 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:59.418201    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:00:59.418497 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:59.418385    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:00:59.418497 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:59.418406    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:00:59.418497 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:59.418425    2561 projected.go:194] Error preparing data for projected volume kube-api-access-4fsct for pod openshift-network-diagnostics/network-check-target-swnch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:59.418497 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:59.418472    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct podName:58051681-235b-4f37-ae82-7b6105f0c847 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:03.418458622 +0000 UTC m=+10.290717039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fsct" (UniqueName: "kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct") pod "network-check-target-swnch" (UID: "58051681-235b-4f37-ae82-7b6105f0c847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:00:59.712960 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:59.710877    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:00:59.712960 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:59.711052    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:00:59.712960 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:00:59.711187    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:00:59.712960 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:00:59.711337    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:01.708109 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:01.708072    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:01.708109 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:01.708099    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:01.708636 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:01.708207    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:01.708636 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:01.708358    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:03.352810 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:03.352768    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:03.353297 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:03.352973    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:03.353297 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:03.353040    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs podName:4fbe1d25-e242-4838-b10c-78c4a9c1196e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:11.353020856 +0000 UTC m=+18.225279268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs") pod "network-metrics-daemon-6glcl" (UID: "4fbe1d25-e242-4838-b10c-78c4a9c1196e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:03.453755 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:03.453715    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:03.453952 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:03.453936    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:03.454022 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:03.453958    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:03.454022 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:03.453971    2561 projected.go:194] Error preparing data for projected volume kube-api-access-4fsct for pod openshift-network-diagnostics/network-check-target-swnch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:03.454126 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:03.454030    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct podName:58051681-235b-4f37-ae82-7b6105f0c847 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:11.454011753 +0000 UTC m=+18.326270183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fsct" (UniqueName: "kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct") pod "network-check-target-swnch" (UID: "58051681-235b-4f37-ae82-7b6105f0c847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:03.708385 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:03.708302    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:03.708531 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:03.708417    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:03.708805 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:03.708774    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:03.708923 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:03.708901    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:05.707676 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:05.707638    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:05.708197 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:05.707774    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:05.708197 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:05.707817    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:05.708197 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:05.707950    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:07.707370 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:07.707330    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:07.707817 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:07.707476    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:07.707817 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:07.707532    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:07.707817 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:07.707666    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:09.707365 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:09.707329    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:09.707365 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:09.707350    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:09.707855 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:09.707436    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:09.707855 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:09.707563    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:11.412292 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:11.412249    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:11.412854 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:11.412430    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:11.412854 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:11.412510    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs podName:4fbe1d25-e242-4838-b10c-78c4a9c1196e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.412487902 +0000 UTC m=+34.284746329 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs") pod "network-metrics-daemon-6glcl" (UID: "4fbe1d25-e242-4838-b10c-78c4a9c1196e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:11.513450 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:11.513414    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:11.513612 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:11.513559    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:11.513612 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:11.513580    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:11.513612 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:11.513594    2561 projected.go:194] Error preparing data for projected volume kube-api-access-4fsct for pod openshift-network-diagnostics/network-check-target-swnch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:11.513747 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:11.513648    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct podName:58051681-235b-4f37-ae82-7b6105f0c847 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.513635599 +0000 UTC m=+34.385894011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fsct" (UniqueName: "kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct") pod "network-check-target-swnch" (UID: "58051681-235b-4f37-ae82-7b6105f0c847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:11.707639 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:11.707559    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:11.707792 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:11.707560    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:11.707792 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:11.707700    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:11.707792 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:11.707761    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:13.708402 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:13.708361    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:13.708742 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:13.708447    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:13.708742 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:13.708461    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:13.708742 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:13.708539    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:14.826386 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.826193    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-52kjn" event={"ID":"37120c41-44fb-484a-b6e6-f6fe6e962658","Type":"ContainerStarted","Data":"0b95950e2828b1713f6d366eb55d41cabd5ca1aa7eb974823b706113d72a6b08"}
May 26 16:01:14.827440 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.827418    2561 generic.go:358] "Generic (PLEG): container finished" podID="e29c2933-d508-4230-9a48-47e2758672d5" containerID="a03bed561f1b6b9a9a63e91cd43ca12bd9c2ccd587b7388a539a56a73761271c" exitCode=0
May 26 16:01:14.827495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.827482    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" event={"ID":"e29c2933-d508-4230-9a48-47e2758672d5","Type":"ContainerDied","Data":"a03bed561f1b6b9a9a63e91cd43ca12bd9c2ccd587b7388a539a56a73761271c"}
May 26 16:01:14.829543 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.829524    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:01:14.830007 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.829914    2561 generic.go:358] "Generic (PLEG): container finished" podID="f9ddde5e-182c-4cd4-b95e-f80ef8fdac39" containerID="a0916af9fcd14e51a7f4d7fc1b4f2f4f09da6c759109f18c9cc40c9c0706c562" exitCode=1
May 26 16:01:14.830054 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.830000    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"1f72439679748b2e2f7128bf2f3d4d18ac053c1948a649fe47bfddbab712886d"}
May 26 16:01:14.830054 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.830025    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"83dcc06bda556260d33098973def25f4aea3f7f213fe6fd78da103ad3ac6922f"}
May 26 16:01:14.830054 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.830039    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerDied","Data":"a0916af9fcd14e51a7f4d7fc1b4f2f4f09da6c759109f18c9cc40c9c0706c562"}
May 26 16:01:14.830054 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.830052    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"6264fe188b288a93709716763265573a441aa8d423998cf25f1154867ec9d57e"}
May 26 16:01:14.831316 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.831296    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c8tfr" event={"ID":"61d768a5-d28e-4ac8-8d64-9a8ab4718e68","Type":"ContainerStarted","Data":"dc0b7ae606fc7f8deedc15d8757c22fb464e70fd0494d51efb7795c057dc434a"}
May 26 16:01:14.832550 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.832526    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhz6k" event={"ID":"c096ec5a-91c6-4ddb-ad74-6d92cf1396f9","Type":"ContainerStarted","Data":"0c46b60e7e411d92d1bcff04237ff3ed1b0c7437fce5cc76cf703a212486a49b"}
May 26 16:01:14.834080 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.834061    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wj5zg" event={"ID":"3f685ebf-142e-4075-888a-d44a456c5812","Type":"ContainerStarted","Data":"a1b3c53fdd2c359bdecd08ebe31d28a4557b126c0db42c7a6cb7ba67d75edb7d"}
May 26 16:01:14.835225 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.835206    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fj8z4" event={"ID":"33633aed-0f35-47bb-9be8-158d6d96a114","Type":"ContainerStarted","Data":"3f8ddb22cd11798aaa70d2c1c637b63543c8b31aaf22161422f96fb59956d81f"}
May 26 16:01:14.836407 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.836389    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl" event={"ID":"56b37ce7-4b56-469f-a8a7-c55247d53a12","Type":"ContainerStarted","Data":"1fe9d842953b9c695dad8e23918e880040dfba60081b21e922bb7b1304c50866"}
May 26 16:01:14.841565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.841519    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-131.ec2.internal" podStartSLOduration=20.84150484 podStartE2EDuration="20.84150484s" podCreationTimestamp="2026-05-26 16:00:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:00:57.811575867 +0000 UTC m=+4.683834308" watchObservedRunningTime="2026-05-26 16:01:14.84150484 +0000 UTC m=+21.713763275"
May 26 16:01:14.842087 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.842053    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-52kjn" podStartSLOduration=3.96278199 podStartE2EDuration="21.842045175s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.263654396 +0000 UTC m=+3.135912808" lastFinishedPulling="2026-05-26 16:01:14.142917564 +0000 UTC m=+21.015175993" observedRunningTime="2026-05-26 16:01:14.841159418 +0000 UTC m=+21.713417852" watchObservedRunningTime="2026-05-26 16:01:14.842045175 +0000 UTC m=+21.714303610"
May 26 16:01:14.857180 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.857131    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jhz6k" podStartSLOduration=3.978453574 podStartE2EDuration="21.857112867s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.266098569 +0000 UTC m=+3.138356984" lastFinishedPulling="2026-05-26 16:01:14.144757863 +0000 UTC m=+21.017016277" observedRunningTime="2026-05-26 16:01:14.857003994 +0000 UTC m=+21.729262429" watchObservedRunningTime="2026-05-26 16:01:14.857112867 +0000 UTC m=+21.729371301"
May 26 16:01:14.875188 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.875140    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-fj8z4" podStartSLOduration=3.978614989 podStartE2EDuration="21.875122227s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.269213475 +0000 UTC m=+3.141471900" lastFinishedPulling="2026-05-26 16:01:14.165720712 +0000 UTC m=+21.037979138" observedRunningTime="2026-05-26 16:01:14.874631981 +0000 UTC m=+21.746890418" watchObservedRunningTime="2026-05-26 16:01:14.875122227 +0000 UTC m=+21.747380662"
May 26 16:01:14.909015 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.908967    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c8tfr" podStartSLOduration=8.66563432 podStartE2EDuration="21.908953244s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.270747556 +0000 UTC m=+3.143005979" lastFinishedPulling="2026-05-26 16:01:09.514066476 +0000 UTC m=+16.386324903" observedRunningTime="2026-05-26 16:01:14.890838365 +0000 UTC m=+21.763096799" watchObservedRunningTime="2026-05-26 16:01:14.908953244 +0000 UTC m=+21.781211678"
May 26 16:01:14.928593 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:14.928548    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wj5zg" podStartSLOduration=4.050661751 podStartE2EDuration="21.928530299s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.265035728 +0000 UTC m=+3.137294145" lastFinishedPulling="2026-05-26 16:01:14.142904266 +0000 UTC m=+21.015162693" observedRunningTime="2026-05-26 16:01:14.908619071 +0000 UTC m=+21.780877505" watchObservedRunningTime="2026-05-26 16:01:14.928530299 +0000 UTC m=+21.800788734"
May 26 16:01:15.672118 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.671956    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 26 16:01:15.707691 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.707663    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:15.707902 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.707693    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:15.707902 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:15.707756    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:15.708035 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:15.707920    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:15.840052 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.839945    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-scmxr" event={"ID":"554d928f-b65a-4f7f-ace1-b4e49f30093c","Type":"ContainerStarted","Data":"f7dfbe9f2be4490b99b9b61d72a35245c06c93a58d93a47def1e697bf46f1977"}
May 26 16:01:15.841862 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.841832    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl" event={"ID":"56b37ce7-4b56-469f-a8a7-c55247d53a12","Type":"ContainerStarted","Data":"d3e74fd12e474808c8a4f97c566a58cdfdfdaf5bd4af4c21f8161550fa21444c"}
May 26 16:01:15.844731 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.844713    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:01:15.845100 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.845073    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"d91a3f2ccfec1d892a9a8d5b4f2dad6a8e51061f67db33a6e90d5ce44bd0ed19"}
May 26 16:01:15.845204 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.845110    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"d5dbff0e6e534721311ffe7bcde824159fda30a2b0a3237a6b343d97b6521fc9"}
May 26 16:01:15.857496 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:15.857449    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-scmxr" podStartSLOduration=4.985085975 podStartE2EDuration="22.857437313s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.270577695 +0000 UTC m=+3.142836110" lastFinishedPulling="2026-05-26 16:01:14.142929032 +0000 UTC m=+21.015187448" observedRunningTime="2026-05-26 16:01:15.856988696 +0000 UTC m=+22.729247131" watchObservedRunningTime="2026-05-26 16:01:15.857437313 +0000 UTC m=+22.729695747"
May 26 16:01:16.420390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:16.420358    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:01:16.648934 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:16.648809    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-26T16:01:15.672112777Z","UUID":"0e9098f8-2698-4664-be1a-b8ad7925c1fd","Handler":null,"Name":"","Endpoint":""}
May 26 16:01:16.650875 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:16.650846    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 26 16:01:16.651006 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:16.650881    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 26 16:01:17.707179 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:17.707145    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:17.707757 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:17.707182    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:17.707757 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:17.707281    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:17.707757 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:17.707389    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:17.851968 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:17.851938    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:01:17.852396 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:17.852366    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"5630c92784703bfe0efaaa9addcfcd1ad4b604e42b8f5fb375e70b85d995772d"}
May 26 16:01:17.854432 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:17.854402    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl" event={"ID":"56b37ce7-4b56-469f-a8a7-c55247d53a12","Type":"ContainerStarted","Data":"9ce7aaecac8cd222233b9a7ff9da84b568cdcf5f71ccd74d97cea5b015a43319"}
May 26 16:01:17.874225 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:17.874176    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6q9bl" podStartSLOduration=4.408654387 podStartE2EDuration="24.874158949s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.263805641 +0000 UTC m=+3.136064053" lastFinishedPulling="2026-05-26 16:01:16.729310186 +0000 UTC m=+23.601568615" observedRunningTime="2026-05-26 16:01:17.873970961 +0000 UTC m=+24.746229416" watchObservedRunningTime="2026-05-26 16:01:17.874158949 +0000 UTC m=+24.746417383"
May 26 16:01:18.178363 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:18.178332    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:01:18.178968 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:18.178943    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:01:18.857657 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:18.857624    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c8tfr"
May 26 16:01:19.707677 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:19.707441    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:19.707832 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:19.707446    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:19.707832 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:19.707783    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:19.707921 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:19.707837    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:19.859676 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:19.859641    2561 generic.go:358] "Generic (PLEG): container finished" podID="e29c2933-d508-4230-9a48-47e2758672d5" containerID="54ef271b9128e14c8da2c4b881cc426cdaaef319ccb917ce7f3b648527fd2d9e" exitCode=0
May 26 16:01:19.860192 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:19.859729    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" event={"ID":"e29c2933-d508-4230-9a48-47e2758672d5","Type":"ContainerDied","Data":"54ef271b9128e14c8da2c4b881cc426cdaaef319ccb917ce7f3b648527fd2d9e"}
May 26 16:01:19.863076 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:19.863058    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:01:19.863505 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:19.863480    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"9616297cba662e6a90cccba5c22e1efb24e23b53e9f7b71b9073c67d520ef8cb"}
May 26 16:01:19.863941 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:19.863928    2561 scope.go:117] "RemoveContainer" containerID="a0916af9fcd14e51a7f4d7fc1b4f2f4f09da6c759109f18c9cc40c9c0706c562"
May 26 16:01:20.868988 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:20.868964    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:01:20.869383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:20.869356    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" event={"ID":"f9ddde5e-182c-4cd4-b95e-f80ef8fdac39","Type":"ContainerStarted","Data":"2387eed18f95e79b4a837174cc184f5fae8d4d957c157791424593b6d50e3cc8"}
May 26 16:01:20.879225 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:20.879081    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:01:20.879225 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:20.879118    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:01:20.879225 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:20.879133    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:01:20.900906 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:20.900850    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:01:20.905352 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:20.905327    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:01:20.913799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:20.913747    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk" podStartSLOduration=9.982428275 podStartE2EDuration="27.913728158s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.258174063 +0000 UTC m=+3.130432476" lastFinishedPulling="2026-05-26 16:01:14.189473931 +0000 UTC m=+21.061732359" observedRunningTime="2026-05-26 16:01:20.913069453 +0000 UTC m=+27.785327887" watchObservedRunningTime="2026-05-26 16:01:20.913728158 +0000 UTC m=+27.785986594"
May 26 16:01:21.121554 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:21.121266    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6glcl"]
May 26 16:01:21.121694 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:21.121632    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:21.121791 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:21.121762    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:21.121944 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:21.121834    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-swnch"]
May 26 16:01:21.122015 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:21.121949    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:21.122072 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:21.122031    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:21.873203 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:21.873169    2561 generic.go:358] "Generic (PLEG): container finished" podID="e29c2933-d508-4230-9a48-47e2758672d5" containerID="c32bf0d2f2c39a3162c84936a5df7aceb829ccffcc63dac18c2690b2cd5c3b8d" exitCode=0
May 26 16:01:21.873544 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:21.873261    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" event={"ID":"e29c2933-d508-4230-9a48-47e2758672d5","Type":"ContainerDied","Data":"c32bf0d2f2c39a3162c84936a5df7aceb829ccffcc63dac18c2690b2cd5c3b8d"}
May 26 16:01:22.707343 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:22.707308    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:22.707343 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:22.707336    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:22.707525 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:22.707419    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:22.707571 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:22.707552    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:23.879789 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:23.879757    2561 generic.go:358] "Generic (PLEG): container finished" podID="e29c2933-d508-4230-9a48-47e2758672d5" containerID="e9b6700c39ca10769cb36dab4747cdd7ab02c1e6f6e4a9836764ced90b0617b2" exitCode=0
May 26 16:01:23.880391 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:23.879838    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" event={"ID":"e29c2933-d508-4230-9a48-47e2758672d5","Type":"ContainerDied","Data":"e9b6700c39ca10769cb36dab4747cdd7ab02c1e6f6e4a9836764ced90b0617b2"}
May 26 16:01:24.707756 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:24.707723    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:24.707943 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:24.707731    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:24.707943 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:24.707844    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:24.708076 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:24.707960    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:26.707959 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:26.707923    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:26.707959 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:26.707950    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:26.708507 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:26.708044    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6glcl" podUID="4fbe1d25-e242-4838-b10c-78c4a9c1196e"
May 26 16:01:26.708507 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:26.708208    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-swnch" podUID="58051681-235b-4f37-ae82-7b6105f0c847"
May 26 16:01:26.987033 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:26.986772    2561 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-133-131.ec2.internal" event="NodeReady"
May 26 16:01:26.987241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:26.987112    2561 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 26 16:01:27.048110 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.048069    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4qxkt"]
May 26 16:01:27.067181 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.067132    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p4xt6"]
May 26 16:01:27.067181 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.067175    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:27.071567 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.071536    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 26 16:01:27.071709 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.071549    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l7p5g\""
May 26 16:01:27.071709 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.071593    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 26 16:01:27.071956 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.071940    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 26 16:01:27.081193 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.081173    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4qxkt"]
May 26 16:01:27.081297 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.081199    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p4xt6"]
May 26 16:01:27.081343 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.081315    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.084782 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.084762    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 26 16:01:27.085044 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.085026    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w25cq\""
May 26 16:01:27.085344 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.085326    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 26 16:01:27.126118 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.126093    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:27.126282 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.126125    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkfs\" (UniqueName: \"kubernetes.io/projected/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-kube-api-access-ktkfs\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:27.227211 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.227179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-config-volume\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.227479 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.227240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.227479 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.227269    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:27.227479 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.227297    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkfs\" (UniqueName: \"kubernetes.io/projected/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-kube-api-access-ktkfs\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:27.227479 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.227367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfpdg\" (UniqueName: \"kubernetes.io/projected/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-kube-api-access-tfpdg\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.227479 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.227398    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:27.227479 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.227427    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-tmp-dir\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.227479 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.227466    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert podName:3b9eef08-2c0d-419e-b42e-5ad0cff81e71 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.727449003 +0000 UTC m=+34.599707427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert") pod "ingress-canary-4qxkt" (UID: "3b9eef08-2c0d-419e-b42e-5ad0cff81e71") : secret "canary-serving-cert" not found
May 26 16:01:27.238165 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.238115    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkfs\" (UniqueName: \"kubernetes.io/projected/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-kube-api-access-ktkfs\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:27.328067 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.328031    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-config-volume\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.328249 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.328086    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.328249 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.328119    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tfpdg\" (UniqueName: \"kubernetes.io/projected/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-kube-api-access-tfpdg\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.328249 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.328174    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-tmp-dir\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.328393 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.328282    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:27.328393 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.328377    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls podName:78c7e49d-53fc-4af3-874c-8a7b6ab588e7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:27.828355771 +0000 UTC m=+34.700614202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls") pod "dns-default-p4xt6" (UID: "78c7e49d-53fc-4af3-874c-8a7b6ab588e7") : secret "dns-default-metrics-tls" not found
May 26 16:01:27.328490 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.328457    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-tmp-dir\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.328725 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.328702    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-config-volume\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.340752 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.340725    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfpdg\" (UniqueName: \"kubernetes.io/projected/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-kube-api-access-tfpdg\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.428908 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.428851    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:27.429077 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.429000    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:27.429077 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.429066    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs podName:4fbe1d25-e242-4838-b10c-78c4a9c1196e nodeName:}" failed. No retries permitted until 2026-05-26 16:01:59.429044534 +0000 UTC m=+66.301302945 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs") pod "network-metrics-daemon-6glcl" (UID: "4fbe1d25-e242-4838-b10c-78c4a9c1196e") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 26 16:01:27.529796 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.529757    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:27.530073 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.529965    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 26 16:01:27.530073 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.529994    2561 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 26 16:01:27.530073 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.530008    2561 projected.go:194] Error preparing data for projected volume kube-api-access-4fsct for pod openshift-network-diagnostics/network-check-target-swnch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:27.530221 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.530124    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct podName:58051681-235b-4f37-ae82-7b6105f0c847 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:59.530099361 +0000 UTC m=+66.402357794 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4fsct" (UniqueName: "kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct") pod "network-check-target-swnch" (UID: "58051681-235b-4f37-ae82-7b6105f0c847") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 26 16:01:27.730553 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.730520    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:27.731063 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.730695    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:27.731063 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.730764    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert podName:3b9eef08-2c0d-419e-b42e-5ad0cff81e71 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.730748648 +0000 UTC m=+35.603007077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert") pod "ingress-canary-4qxkt" (UID: "3b9eef08-2c0d-419e-b42e-5ad0cff81e71") : secret "canary-serving-cert" not found
May 26 16:01:27.831719 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:27.831629    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:27.831912 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.831804    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:27.831912 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:27.831883    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls podName:78c7e49d-53fc-4af3-874c-8a7b6ab588e7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:28.831862373 +0000 UTC m=+35.704120791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls") pod "dns-default-p4xt6" (UID: "78c7e49d-53fc-4af3-874c-8a7b6ab588e7") : secret "dns-default-metrics-tls" not found
May 26 16:01:28.708054 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.708017    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:28.708237 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.708017    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:28.711269 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.711234    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:28.712208 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.712178    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsz4f\""
May 26 16:01:28.712208 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.712180    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:28.712400 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.712180    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:28.712400 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.712337    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf4w7\""
May 26 16:01:28.738775 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.738756    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:28.739105 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:28.738857    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:28.739105 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:28.738940    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert podName:3b9eef08-2c0d-419e-b42e-5ad0cff81e71 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:30.738925317 +0000 UTC m=+37.611183746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert") pod "ingress-canary-4qxkt" (UID: "3b9eef08-2c0d-419e-b42e-5ad0cff81e71") : secret "canary-serving-cert" not found
May 26 16:01:28.839480 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:28.839445    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:28.839667 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:28.839607    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:28.839737 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:28.839683    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls podName:78c7e49d-53fc-4af3-874c-8a7b6ab588e7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:30.839662574 +0000 UTC m=+37.711921007 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls") pod "dns-default-p4xt6" (UID: "78c7e49d-53fc-4af3-874c-8a7b6ab588e7") : secret "dns-default-metrics-tls" not found
May 26 16:01:30.755137 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:30.755111    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:30.755441 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:30.755211    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:30.755441 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:30.755268    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert podName:3b9eef08-2c0d-419e-b42e-5ad0cff81e71 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:34.755254718 +0000 UTC m=+41.627513130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert") pod "ingress-canary-4qxkt" (UID: "3b9eef08-2c0d-419e-b42e-5ad0cff81e71") : secret "canary-serving-cert" not found
May 26 16:01:30.855584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:30.855547    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:30.855712 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:30.855673    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:30.855765 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:30.855729    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls podName:78c7e49d-53fc-4af3-874c-8a7b6ab588e7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:34.855713587 +0000 UTC m=+41.727972001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls") pod "dns-default-p4xt6" (UID: "78c7e49d-53fc-4af3-874c-8a7b6ab588e7") : secret "dns-default-metrics-tls" not found
May 26 16:01:30.895468 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:30.895440    2561 generic.go:358] "Generic (PLEG): container finished" podID="e29c2933-d508-4230-9a48-47e2758672d5" containerID="29e9a5c9582d25b4b87f3c21b7ee7e30977783a3de631da68e629349b33dea0d" exitCode=0
May 26 16:01:30.895606 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:30.895493    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" event={"ID":"e29c2933-d508-4230-9a48-47e2758672d5","Type":"ContainerDied","Data":"29e9a5c9582d25b4b87f3c21b7ee7e30977783a3de631da68e629349b33dea0d"}
May 26 16:01:31.900028 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:31.899995    2561 generic.go:358] "Generic (PLEG): container finished" podID="e29c2933-d508-4230-9a48-47e2758672d5" containerID="8c071f58e0e4377f42103a75c1543d7e28dcf140b4c324eff2ff489f5bd1125f" exitCode=0
May 26 16:01:31.900413 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:31.900048    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" event={"ID":"e29c2933-d508-4230-9a48-47e2758672d5","Type":"ContainerDied","Data":"8c071f58e0e4377f42103a75c1543d7e28dcf140b4c324eff2ff489f5bd1125f"}
May 26 16:01:32.904723 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:32.904686    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" event={"ID":"e29c2933-d508-4230-9a48-47e2758672d5","Type":"ContainerStarted","Data":"2be6e38c94e5ccec04ef27ebcf8142a9a9bc68fa74ce974fbc0187254c10a4f3"}
May 26 16:01:32.931383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:32.931311    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vn5k7" podStartSLOduration=6.366023534 podStartE2EDuration="39.931299308s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:00:56.259169404 +0000 UTC m=+3.131427830" lastFinishedPulling="2026-05-26 16:01:29.824445191 +0000 UTC m=+36.696703604" observedRunningTime="2026-05-26 16:01:32.930298221 +0000 UTC m=+39.802556656" watchObservedRunningTime="2026-05-26 16:01:32.931299308 +0000 UTC m=+39.803557742"
May 26 16:01:34.782007 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:34.781975    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:34.782347 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:34.782081    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:34.782347 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:34.782133    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert podName:3b9eef08-2c0d-419e-b42e-5ad0cff81e71 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:42.782117887 +0000 UTC m=+49.654376302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert") pod "ingress-canary-4qxkt" (UID: "3b9eef08-2c0d-419e-b42e-5ad0cff81e71") : secret "canary-serving-cert" not found
May 26 16:01:34.882563 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:34.882530    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:34.882747 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:34.882686    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:34.882793 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:34.882749    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls podName:78c7e49d-53fc-4af3-874c-8a7b6ab588e7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:42.882734545 +0000 UTC m=+49.754992957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls") pod "dns-default-p4xt6" (UID: "78c7e49d-53fc-4af3-874c-8a7b6ab588e7") : secret "dns-default-metrics-tls" not found
May 26 16:01:42.836284 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:42.836247    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:42.836707 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:42.836372    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:42.836707 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:42.836429    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert podName:3b9eef08-2c0d-419e-b42e-5ad0cff81e71 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:58.836412606 +0000 UTC m=+65.708671039 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert") pod "ingress-canary-4qxkt" (UID: "3b9eef08-2c0d-419e-b42e-5ad0cff81e71") : secret "canary-serving-cert" not found
May 26 16:01:42.937137 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:42.937108    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:42.937270 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:42.937207    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:42.937270 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:42.937253    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls podName:78c7e49d-53fc-4af3-874c-8a7b6ab588e7 nodeName:}" failed. No retries permitted until 2026-05-26 16:01:58.937239938 +0000 UTC m=+65.809498351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls") pod "dns-default-p4xt6" (UID: "78c7e49d-53fc-4af3-874c-8a7b6ab588e7") : secret "dns-default-metrics-tls" not found
May 26 16:01:52.885240 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:52.885211    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jslmk"
May 26 16:01:58.847968 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:58.847929    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:01:58.848345 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:58.848061    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:01:58.848345 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:58.848125    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert podName:3b9eef08-2c0d-419e-b42e-5ad0cff81e71 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:30.848108959 +0000 UTC m=+97.720367372 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert") pod "ingress-canary-4qxkt" (UID: "3b9eef08-2c0d-419e-b42e-5ad0cff81e71") : secret "canary-serving-cert" not found
May 26 16:01:58.948200 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:58.948168    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:01:58.948335 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:58.948312    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:01:58.948393 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:58.948382    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls podName:78c7e49d-53fc-4af3-874c-8a7b6ab588e7 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:30.948364362 +0000 UTC m=+97.820622797 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls") pod "dns-default-p4xt6" (UID: "78c7e49d-53fc-4af3-874c-8a7b6ab588e7") : secret "dns-default-metrics-tls" not found
May 26 16:01:59.451474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.451440    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:01:59.454285 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.454267    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 26 16:01:59.462421 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:59.462401    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
May 26 16:01:59.462481 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:01:59.462472    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs podName:4fbe1d25-e242-4838-b10c-78c4a9c1196e nodeName:}" failed. No retries permitted until 2026-05-26 16:03:03.462455134 +0000 UTC m=+130.334713546 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs") pod "network-metrics-daemon-6glcl" (UID: "4fbe1d25-e242-4838-b10c-78c4a9c1196e") : secret "metrics-daemon-secret" not found
May 26 16:01:59.552775 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.552742    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:59.555744 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.555727    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 26 16:01:59.566138 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.566121    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 26 16:01:59.577361 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.577330    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fsct\" (UniqueName: \"kubernetes.io/projected/58051681-235b-4f37-ae82-7b6105f0c847-kube-api-access-4fsct\") pod \"network-check-target-swnch\" (UID: \"58051681-235b-4f37-ae82-7b6105f0c847\") " pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:59.628241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.628217    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nf4w7\""
May 26 16:01:59.635417 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.635398    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:01:59.782517 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.782485    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-swnch"]
May 26 16:01:59.787011 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:01:59.786980    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58051681_235b_4f37_ae82_7b6105f0c847.slice/crio-40805aa94d13b1984d5bd54c039e4559d7fd318803194f3fe3d2f1aeaabd2e26 WatchSource:0}: Error finding container 40805aa94d13b1984d5bd54c039e4559d7fd318803194f3fe3d2f1aeaabd2e26: Status 404 returned error can't find the container with id 40805aa94d13b1984d5bd54c039e4559d7fd318803194f3fe3d2f1aeaabd2e26
May 26 16:01:59.955236 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:01:59.955194    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-swnch" event={"ID":"58051681-235b-4f37-ae82-7b6105f0c847","Type":"ContainerStarted","Data":"40805aa94d13b1984d5bd54c039e4559d7fd318803194f3fe3d2f1aeaabd2e26"}
May 26 16:02:02.962022 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:02.961975    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-swnch" event={"ID":"58051681-235b-4f37-ae82-7b6105f0c847","Type":"ContainerStarted","Data":"66648eb30be42590d08a5fb968c0839602bf618a8a3ddb8d3cc2a67f8a743904"}
May 26 16:02:02.962417 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:02.962214    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:02:02.980582 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:02.980539    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-swnch" podStartSLOduration=67.321047299 podStartE2EDuration="1m9.98052641s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:01:59.788858584 +0000 UTC m=+66.661116999" lastFinishedPulling="2026-05-26 16:02:02.448337683 +0000 UTC m=+69.320596110" observedRunningTime="2026-05-26 16:02:02.97963132 +0000 UTC m=+69.851889755" watchObservedRunningTime="2026-05-26 16:02:02.98052641 +0000 UTC m=+69.852784866"
May 26 16:02:30.860480 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:30.860447    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:02:30.860925 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:30.860599    2561 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found
May 26 16:02:30.860925 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:30.860676    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert podName:3b9eef08-2c0d-419e-b42e-5ad0cff81e71 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:34.860660568 +0000 UTC m=+161.732918980 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert") pod "ingress-canary-4qxkt" (UID: "3b9eef08-2c0d-419e-b42e-5ad0cff81e71") : secret "canary-serving-cert" not found
May 26 16:02:30.961201 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:30.961117    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:02:30.961330 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:30.961231    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
May 26 16:02:30.961330 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:30.961295    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls podName:78c7e49d-53fc-4af3-874c-8a7b6ab588e7 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:34.961280815 +0000 UTC m=+161.833539226 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls") pod "dns-default-p4xt6" (UID: "78c7e49d-53fc-4af3-874c-8a7b6ab588e7") : secret "dns-default-metrics-tls" not found
May 26 16:02:33.966265 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:33.966236    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-swnch"
May 26 16:02:42.595469 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.595436    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"]
May 26 16:02:42.597223 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.597205    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d8488b484-56l4v"]
May 26 16:02:42.597375 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.597355    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.598911 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.598879    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.599913 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.599877    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
May 26 16:02:42.599995 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.599884    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wtpkj\""
May 26 16:02:42.601136 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.601116    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 26 16:02:42.601302 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.601280    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 26 16:02:42.601956 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.601940    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
May 26 16:02:42.601956 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.601958    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
May 26 16:02:42.602096 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.602013    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
May 26 16:02:42.602096 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.602016    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
May 26 16:02:42.602096 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.602056    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
May 26 16:02:42.602096 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.602071    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gjmpj\""
May 26 16:02:42.602235 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.602112    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
May 26 16:02:42.602389 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.602376    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
May 26 16:02:42.612436 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.612415    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"]
May 26 16:02:42.613046 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.613026    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d8488b484-56l4v"]
May 26 16:02:42.639118 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.639095    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-default-certificate\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.639222 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.639123    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-stats-auth\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.639222 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.639142    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vckwn\" (UniqueName: \"kubernetes.io/projected/6c9859ce-6c0b-4dad-a6e9-4b672695357a-kube-api-access-vckwn\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.639222 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.639179    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tgpz\" (UniqueName: \"kubernetes.io/projected/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-kube-api-access-4tgpz\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.639222 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.639220    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.639350 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.639240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.639350 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.639285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.639350 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.639300    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.718273 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.718237    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"]
May 26 16:02:42.720047 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.720032    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:42.726984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.726955    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v44xz\""
May 26 16:02:42.727090 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.727048    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
May 26 16:02:42.727156 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.727114    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
May 26 16:02:42.727870 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.727857    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
May 26 16:02:42.730847 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.730830    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"]
May 26 16:02:42.740316 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740295    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79nck\" (UniqueName: \"kubernetes.io/projected/8d3e3ee6-0d93-4860-b465-1438011c573b-kube-api-access-79nck\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:42.740431 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740343    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.740431 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740374    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.740550 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:42.740496    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:02:42.740550 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740505    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-default-certificate\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.740644 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:42.740557    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:43.240540921 +0000 UTC m=+110.112799333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : secret "router-metrics-certs-default" not found
May 26 16:02:42.740644 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:42.740574    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:43.24056882 +0000 UTC m=+110.112827232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : configmap references non-existent config key: service-ca.crt
May 26 16:02:42.740644 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740552    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-stats-auth\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.740644 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740611    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vckwn\" (UniqueName: \"kubernetes.io/projected/6c9859ce-6c0b-4dad-a6e9-4b672695357a-kube-api-access-vckwn\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.740860 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740646    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4tgpz\" (UniqueName: \"kubernetes.io/projected/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-kube-api-access-4tgpz\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.740860 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740667    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:42.740860 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740714    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.740860 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.740774    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.741082 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:42.740943    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:02:42.741082 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:42.741027    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls podName:2bae3ffc-8cbc-46c6-b79f-f0bd40f70940 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:43.24101151 +0000 UTC m=+110.113269927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7dmc6" (UID: "2bae3ffc-8cbc-46c6-b79f-f0bd40f70940") : secret "cluster-monitoring-operator-tls" not found
May 26 16:02:42.741442 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.741421    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.742898 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.742870    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-stats-auth\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.742985 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.742969    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-default-certificate\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.758778 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.758747    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tgpz\" (UniqueName: \"kubernetes.io/projected/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-kube-api-access-4tgpz\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:42.760463 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.760447    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vckwn\" (UniqueName: \"kubernetes.io/projected/6c9859ce-6c0b-4dad-a6e9-4b672695357a-kube-api-access-vckwn\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:42.811401 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.811374    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"]
May 26 16:02:42.814759 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.814426    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.817461 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.817440    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
May 26 16:02:42.817576 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.817443    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
May 26 16:02:42.817711 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.817696    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
May 26 16:02:42.817760 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.817709    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
May 26 16:02:42.817800 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.817760    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wzz4f\""
May 26 16:02:42.823337 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.823316    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"]
May 26 16:02:42.841629 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.841606    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:42.841733 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.841652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfn5j\" (UniqueName: \"kubernetes.io/projected/3d007b98-2dc2-43a6-9083-5cdbda352f5e-kube-api-access-vfn5j\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.841733 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.841689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d007b98-2dc2-43a6-9083-5cdbda352f5e-serving-cert\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.841733 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.841720    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-79nck\" (UniqueName: \"kubernetes.io/projected/8d3e3ee6-0d93-4860-b465-1438011c573b-kube-api-access-79nck\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:42.841831 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:42.841746    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:02:42.841831 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:42.841796    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls podName:8d3e3ee6-0d93-4860-b465-1438011c573b nodeName:}" failed. No retries permitted until 2026-05-26 16:02:43.341781543 +0000 UTC m=+110.214039955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls") pod "cluster-samples-operator-697587599-pb6f4" (UID: "8d3e3ee6-0d93-4860-b465-1438011c573b") : secret "samples-operator-tls" not found
May 26 16:02:42.841923 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.841841    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d007b98-2dc2-43a6-9083-5cdbda352f5e-config\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.850549 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.850489    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79nck\" (UniqueName: \"kubernetes.io/projected/8d3e3ee6-0d93-4860-b465-1438011c573b-kube-api-access-79nck\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:42.942872 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.942836    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vfn5j\" (UniqueName: \"kubernetes.io/projected/3d007b98-2dc2-43a6-9083-5cdbda352f5e-kube-api-access-vfn5j\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.943022 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.942882    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d007b98-2dc2-43a6-9083-5cdbda352f5e-serving-cert\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.943022 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.942966    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d007b98-2dc2-43a6-9083-5cdbda352f5e-config\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.943415 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.943384    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d007b98-2dc2-43a6-9083-5cdbda352f5e-config\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.945123 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.945104    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d007b98-2dc2-43a6-9083-5cdbda352f5e-serving-cert\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:42.951089 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:42.951072    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfn5j\" (UniqueName: \"kubernetes.io/projected/3d007b98-2dc2-43a6-9083-5cdbda352f5e-kube-api-access-vfn5j\") pod \"service-ca-operator-57b5b68b8-qwmwv\" (UID: \"3d007b98-2dc2-43a6-9083-5cdbda352f5e\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:43.124520 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:43.124419    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"
May 26 16:02:43.240951 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:43.240919    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv"]
May 26 16:02:43.244511 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:02:43.244486    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d007b98_2dc2_43a6_9083_5cdbda352f5e.slice/crio-95054cdd9a1062e214db4e39098f77a10eb03b29432c4ad9f775dc5bfe71376d WatchSource:0}: Error finding container 95054cdd9a1062e214db4e39098f77a10eb03b29432c4ad9f775dc5bfe71376d: Status 404 returned error can't find the container with id 95054cdd9a1062e214db4e39098f77a10eb03b29432c4ad9f775dc5bfe71376d
May 26 16:02:43.244789 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:43.244770    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:43.244865 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:43.244847    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:43.245016 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:43.244876    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:43.245016 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:43.244933    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:02:43.245016 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:43.245002    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls podName:2bae3ffc-8cbc-46c6-b79f-f0bd40f70940 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:44.244982827 +0000 UTC m=+111.117241253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7dmc6" (UID: "2bae3ffc-8cbc-46c6-b79f-f0bd40f70940") : secret "cluster-monitoring-operator-tls" not found
May 26 16:02:43.245149 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:43.245019    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:44.245010858 +0000 UTC m=+111.117269269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : configmap references non-existent config key: service-ca.crt
May 26 16:02:43.245149 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:43.245023    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:02:43.245149 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:43.245070    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:44.245053605 +0000 UTC m=+111.117312032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : secret "router-metrics-certs-default" not found
May 26 16:02:43.345574 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:43.345540    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:43.345718 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:43.345674    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:02:43.345757 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:43.345745    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls podName:8d3e3ee6-0d93-4860-b465-1438011c573b nodeName:}" failed. No retries permitted until 2026-05-26 16:02:44.34573276 +0000 UTC m=+111.217991178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls") pod "cluster-samples-operator-697587599-pb6f4" (UID: "8d3e3ee6-0d93-4860-b465-1438011c573b") : secret "samples-operator-tls" not found
May 26 16:02:44.041528 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:44.041485    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv" event={"ID":"3d007b98-2dc2-43a6-9083-5cdbda352f5e","Type":"ContainerStarted","Data":"95054cdd9a1062e214db4e39098f77a10eb03b29432c4ad9f775dc5bfe71376d"}
May 26 16:02:44.251621 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:44.251587    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:44.251821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:44.251668    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:44.251821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:44.251694    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:44.251821 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:44.251768    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:02:44.251991 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:44.251840    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:02:44.251991 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:44.251853    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:46.251830506 +0000 UTC m=+113.124088921 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : configmap references non-existent config key: service-ca.crt
May 26 16:02:44.251991 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:44.251880    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls podName:2bae3ffc-8cbc-46c6-b79f-f0bd40f70940 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:46.25187182 +0000 UTC m=+113.124130232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7dmc6" (UID: "2bae3ffc-8cbc-46c6-b79f-f0bd40f70940") : secret "cluster-monitoring-operator-tls" not found
May 26 16:02:44.251991 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:44.251912    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:46.25190233 +0000 UTC m=+113.124160760 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : secret "router-metrics-certs-default" not found
May 26 16:02:44.352218 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:44.352137    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:44.352418 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:44.352299    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:02:44.352418 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:44.352372    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls podName:8d3e3ee6-0d93-4860-b465-1438011c573b nodeName:}" failed. No retries permitted until 2026-05-26 16:02:46.352355766 +0000 UTC m=+113.224614198 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls") pod "cluster-samples-operator-697587599-pb6f4" (UID: "8d3e3ee6-0d93-4860-b465-1438011c573b") : secret "samples-operator-tls" not found
May 26 16:02:46.046369 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:46.046336    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv" event={"ID":"3d007b98-2dc2-43a6-9083-5cdbda352f5e","Type":"ContainerStarted","Data":"21dfaefe5dfd7ac36e2e179f9c2b2519ac8e90eaa99b3212de9232f7163c3ecc"}
May 26 16:02:46.063718 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:46.063669    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv" podStartSLOduration=1.8487099649999998 podStartE2EDuration="4.06365638s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="2026-05-26 16:02:43.246155515 +0000 UTC m=+110.118413927" lastFinishedPulling="2026-05-26 16:02:45.461101917 +0000 UTC m=+112.333360342" observedRunningTime="2026-05-26 16:02:46.062582062 +0000 UTC m=+112.934840486" watchObservedRunningTime="2026-05-26 16:02:46.06365638 +0000 UTC m=+112.935914813"
May 26 16:02:46.267808 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:46.267774    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:46.268005 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:46.267861    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:46.268005 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:46.267936    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:02:46.268005 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:46.267944    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:46.268005 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:46.267996    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:50.267977481 +0000 UTC m=+117.140235907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : secret "router-metrics-certs-default" not found
May 26 16:02:46.268166 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:46.268027    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:02:46.268166 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:46.268037    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:50.268027175 +0000 UTC m=+117.140285594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : configmap references non-existent config key: service-ca.crt
May 26 16:02:46.268166 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:46.268104    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls podName:2bae3ffc-8cbc-46c6-b79f-f0bd40f70940 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:50.268086135 +0000 UTC m=+117.140344553 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7dmc6" (UID: "2bae3ffc-8cbc-46c6-b79f-f0bd40f70940") : secret "cluster-monitoring-operator-tls" not found
May 26 16:02:46.368765 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:46.368676    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:46.368927 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:46.368827    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:02:46.368927 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:46.368915    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls podName:8d3e3ee6-0d93-4860-b465-1438011c573b nodeName:}" failed. No retries permitted until 2026-05-26 16:02:50.368877394 +0000 UTC m=+117.241135809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls") pod "cluster-samples-operator-697587599-pb6f4" (UID: "8d3e3ee6-0d93-4860-b465-1438011c573b") : secret "samples-operator-tls" not found
May 26 16:02:49.024457 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.024422    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-sr7cm"]
May 26 16:02:49.026443 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.026425    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.028917 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.028875    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
May 26 16:02:49.029741 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.029720    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
May 26 16:02:49.029878 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.029778    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-v7hml\""
May 26 16:02:49.030012 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.029998    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
May 26 16:02:49.030102 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.030013    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
May 26 16:02:49.041486 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.041468    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-sr7cm"]
May 26 16:02:49.090717 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.090689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/563d1b38-8144-4d95-9c52-266f93222802-signing-cabundle\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.090819 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.090744    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/563d1b38-8144-4d95-9c52-266f93222802-signing-key\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.090819 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.090772    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9v5l\" (UniqueName: \"kubernetes.io/projected/563d1b38-8144-4d95-9c52-266f93222802-kube-api-access-z9v5l\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.191837 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.191808    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/563d1b38-8144-4d95-9c52-266f93222802-signing-cabundle\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.191959 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.191866    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/563d1b38-8144-4d95-9c52-266f93222802-signing-key\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.192012 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.191994    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z9v5l\" (UniqueName: \"kubernetes.io/projected/563d1b38-8144-4d95-9c52-266f93222802-kube-api-access-z9v5l\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.192473 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.192456    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/563d1b38-8144-4d95-9c52-266f93222802-signing-cabundle\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.194258 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.194239    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/563d1b38-8144-4d95-9c52-266f93222802-signing-key\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.200274 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.200248    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9v5l\" (UniqueName: \"kubernetes.io/projected/563d1b38-8144-4d95-9c52-266f93222802-kube-api-access-z9v5l\") pod \"service-ca-67d667c7d-sr7cm\" (UID: \"563d1b38-8144-4d95-9c52-266f93222802\") " pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.335061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.334983    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-67d667c7d-sr7cm"
May 26 16:02:49.394930 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.394859    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52kjn_37120c41-44fb-484a-b6e6-f6fe6e962658/dns-node-resolver/0.log"
May 26 16:02:49.454703 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:49.454675    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-67d667c7d-sr7cm"]
May 26 16:02:49.459937 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:02:49.459904    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563d1b38_8144_4d95_9c52_266f93222802.slice/crio-7baec014a6cc4a242168bdf1a6be1359c0a0e30019d7146958f664fcc30b4be1 WatchSource:0}: Error finding container 7baec014a6cc4a242168bdf1a6be1359c0a0e30019d7146958f664fcc30b4be1: Status 404 returned error can't find the container with id 7baec014a6cc4a242168bdf1a6be1359c0a0e30019d7146958f664fcc30b4be1
May 26 16:02:50.055439 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:50.055402    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-sr7cm" event={"ID":"563d1b38-8144-4d95-9c52-266f93222802","Type":"ContainerStarted","Data":"2d3f0db14e9e5ce741bba0c6ee7fe55990b9d04c8392ed85df7b97fcb8fc99e8"}
May 26 16:02:50.055439 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:50.055437    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-67d667c7d-sr7cm" event={"ID":"563d1b38-8144-4d95-9c52-266f93222802","Type":"ContainerStarted","Data":"7baec014a6cc4a242168bdf1a6be1359c0a0e30019d7146958f664fcc30b4be1"}
May 26 16:02:50.075048 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:50.075004    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-67d667c7d-sr7cm" podStartSLOduration=1.074989343 podStartE2EDuration="1.074989343s" podCreationTimestamp="2026-05-26 16:02:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:02:50.074263288 +0000 UTC m=+116.946521723" watchObservedRunningTime="2026-05-26 16:02:50.074989343 +0000 UTC m=+116.947247777"
May 26 16:02:50.300227 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:50.300168    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:50.300431 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:50.300282    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:50.300431 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:50.300329    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:50.300431 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:50.300345    2561 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
May 26 16:02:50.300431 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:50.300426    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls podName:2bae3ffc-8cbc-46c6-b79f-f0bd40f70940 nodeName:}" failed. No retries permitted until 2026-05-26 16:02:58.300397227 +0000 UTC m=+125.172655656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-7dmc6" (UID: "2bae3ffc-8cbc-46c6-b79f-f0bd40f70940") : secret "cluster-monitoring-operator-tls" not found
May 26 16:02:50.300653 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:50.300509    2561 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
May 26 16:02:50.300653 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:50.300523    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:58.300504419 +0000 UTC m=+125.172762831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : configmap references non-existent config key: service-ca.crt
May 26 16:02:50.300653 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:50.300576    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs podName:6c9859ce-6c0b-4dad-a6e9-4b672695357a nodeName:}" failed. No retries permitted until 2026-05-26 16:02:58.300545105 +0000 UTC m=+125.172803516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs") pod "router-default-7d8488b484-56l4v" (UID: "6c9859ce-6c0b-4dad-a6e9-4b672695357a") : secret "router-metrics-certs-default" not found
May 26 16:02:50.401771 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:50.401667    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:50.401954 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:50.401854    2561 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
May 26 16:02:50.402013 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:02:50.401962    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls podName:8d3e3ee6-0d93-4860-b465-1438011c573b nodeName:}" failed. No retries permitted until 2026-05-26 16:02:58.401940385 +0000 UTC m=+125.274198815 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls") pod "cluster-samples-operator-697587599-pb6f4" (UID: "8d3e3ee6-0d93-4860-b465-1438011c573b") : secret "samples-operator-tls" not found
May 26 16:02:50.796070 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:50.796032    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wj5zg_3f685ebf-142e-4075-888a-d44a456c5812/node-ca/0.log"
May 26 16:02:58.365830 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.365770    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:58.366338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.365849    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:58.366338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.365871    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:58.366488 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.366465    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c9859ce-6c0b-4dad-a6e9-4b672695357a-service-ca-bundle\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:58.368170 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.368147    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9859ce-6c0b-4dad-a6e9-4b672695357a-metrics-certs\") pod \"router-default-7d8488b484-56l4v\" (UID: \"6c9859ce-6c0b-4dad-a6e9-4b672695357a\") " pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:58.368267 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.368244    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bae3ffc-8cbc-46c6-b79f-f0bd40f70940-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-7dmc6\" (UID: \"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:58.466565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.466530    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:58.468760 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.468737    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/8d3e3ee6-0d93-4860-b465-1438011c573b-samples-operator-tls\") pod \"cluster-samples-operator-697587599-pb6f4\" (UID: \"8d3e3ee6-0d93-4860-b465-1438011c573b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:58.510212 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.510189    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wtpkj\""
May 26 16:02:58.515786 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.515764    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gjmpj\""
May 26 16:02:58.517740 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.517722    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"
May 26 16:02:58.524434 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.524408    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:58.631882 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.631792    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-v44xz\""
May 26 16:02:58.639836 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.639812    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"
May 26 16:02:58.645937 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.645875    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6"]
May 26 16:02:58.650412 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:02:58.650370    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bae3ffc_8cbc_46c6_b79f_f0bd40f70940.slice/crio-870cc702c2011b398d800ec9dd2149b43a05622e7820da6e8aa1e113f329b0d4 WatchSource:0}: Error finding container 870cc702c2011b398d800ec9dd2149b43a05622e7820da6e8aa1e113f329b0d4: Status 404 returned error can't find the container with id 870cc702c2011b398d800ec9dd2149b43a05622e7820da6e8aa1e113f329b0d4
May 26 16:02:58.663869 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.663839    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d8488b484-56l4v"]
May 26 16:02:58.666125 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:02:58.666103    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9859ce_6c0b_4dad_a6e9_4b672695357a.slice/crio-cbbc9f9a8778479e0688939fa8e43cab17f20122e0f3b195472cc6dad6b91054 WatchSource:0}: Error finding container cbbc9f9a8778479e0688939fa8e43cab17f20122e0f3b195472cc6dad6b91054: Status 404 returned error can't find the container with id cbbc9f9a8778479e0688939fa8e43cab17f20122e0f3b195472cc6dad6b91054
May 26 16:02:58.779879 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:58.779845    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4"]
May 26 16:02:59.074224 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:59.074178    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d8488b484-56l4v" event={"ID":"6c9859ce-6c0b-4dad-a6e9-4b672695357a","Type":"ContainerStarted","Data":"1143cc384e5b13b33218d31ac69f2b07d532a2a4aa326b7044164aa8731582fb"}
May 26 16:02:59.074408 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:59.074230    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d8488b484-56l4v" event={"ID":"6c9859ce-6c0b-4dad-a6e9-4b672695357a","Type":"ContainerStarted","Data":"cbbc9f9a8778479e0688939fa8e43cab17f20122e0f3b195472cc6dad6b91054"}
May 26 16:02:59.075446 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:59.075419    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6" event={"ID":"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940","Type":"ContainerStarted","Data":"870cc702c2011b398d800ec9dd2149b43a05622e7820da6e8aa1e113f329b0d4"}
May 26 16:02:59.076345 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:59.076321    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4" event={"ID":"8d3e3ee6-0d93-4860-b465-1438011c573b","Type":"ContainerStarted","Data":"9467b96a98175a292dfd8a8ea2715f9a825b6c05d566336b52a9f0b3505a3392"}
May 26 16:02:59.096696 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:59.096656    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-7d8488b484-56l4v" podStartSLOduration=17.09664444 podStartE2EDuration="17.09664444s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:02:59.096015436 +0000 UTC m=+125.968273892" watchObservedRunningTime="2026-05-26 16:02:59.09664444 +0000 UTC m=+125.968902874"
May 26 16:02:59.524826 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:59.524775    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:02:59.527800 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:02:59.527776    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:03:00.079093 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:00.079059    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:03:00.080485 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:00.080458    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d8488b484-56l4v"
May 26 16:03:01.083603 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:01.083514    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4" event={"ID":"8d3e3ee6-0d93-4860-b465-1438011c573b","Type":"ContainerStarted","Data":"1ed55929ec881628f72af952b8e125917a4c3b5cfb44c4d55bd3ae2a440c1988"}
May 26 16:03:01.083603 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:01.083557    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4" event={"ID":"8d3e3ee6-0d93-4860-b465-1438011c573b","Type":"ContainerStarted","Data":"f72249a792af7f48a2da695aa14124676e77ad0971d44dc8d5b86f1346cb6a99"}
May 26 16:03:01.085132 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:01.085089    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6" event={"ID":"2bae3ffc-8cbc-46c6-b79f-f0bd40f70940","Type":"ContainerStarted","Data":"06f13cb3b7df6a5131140ff4edbe0fa7f7bbfa38ae7f2d436a28f0e1d2cb0b07"}
May 26 16:03:01.109453 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:01.109391    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-pb6f4" podStartSLOduration=17.132125336 podStartE2EDuration="19.109372017s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="2026-05-26 16:02:58.818928747 +0000 UTC m=+125.691187158" lastFinishedPulling="2026-05-26 16:03:00.796175415 +0000 UTC m=+127.668433839" observedRunningTime="2026-05-26 16:03:01.108839147 +0000 UTC m=+127.981097581" watchObservedRunningTime="2026-05-26 16:03:01.109372017 +0000 UTC m=+127.981630453"
May 26 16:03:01.133749 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:01.133694    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-7dmc6" podStartSLOduration=16.991654269 podStartE2EDuration="19.133682033s" podCreationTimestamp="2026-05-26 16:02:42 +0000 UTC" firstStartedPulling="2026-05-26 16:02:58.652293777 +0000 UTC m=+125.524552193" lastFinishedPulling="2026-05-26 16:03:00.794321542 +0000 UTC m=+127.666579957" observedRunningTime="2026-05-26 16:03:01.133035949 +0000 UTC m=+128.005294384" watchObservedRunningTime="2026-05-26 16:03:01.133682033 +0000 UTC m=+128.005940466"
May 26 16:03:03.508843 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:03.508803    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:03:03.511048 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:03.511027    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4fbe1d25-e242-4838-b10c-78c4a9c1196e-metrics-certs\") pod \"network-metrics-daemon-6glcl\" (UID: \"4fbe1d25-e242-4838-b10c-78c4a9c1196e\") " pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:03:03.522704 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:03.522684    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jsz4f\""
May 26 16:03:03.530626 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:03.530604    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6glcl"
May 26 16:03:03.651393 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:03.651367    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6glcl"]
May 26 16:03:03.654584 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:03.654561    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbe1d25_e242_4838_b10c_78c4a9c1196e.slice/crio-9293454894eb61c759bc336709befccec7de2607f1edfd423ab0489899d03bf0 WatchSource:0}: Error finding container 9293454894eb61c759bc336709befccec7de2607f1edfd423ab0489899d03bf0: Status 404 returned error can't find the container with id 9293454894eb61c759bc336709befccec7de2607f1edfd423ab0489899d03bf0
May 26 16:03:04.095606 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:04.095559    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6glcl" event={"ID":"4fbe1d25-e242-4838-b10c-78c4a9c1196e","Type":"ContainerStarted","Data":"9293454894eb61c759bc336709befccec7de2607f1edfd423ab0489899d03bf0"}
May 26 16:03:05.099861 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:05.099824    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6glcl" event={"ID":"4fbe1d25-e242-4838-b10c-78c4a9c1196e","Type":"ContainerStarted","Data":"e41c486c976b751dcaaf88346d2c2b70710860c6aa851babcdbb35d07024a3e6"}
May 26 16:03:05.099861 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:05.099863    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6glcl" event={"ID":"4fbe1d25-e242-4838-b10c-78c4a9c1196e","Type":"ContainerStarted","Data":"a1cc8ffee0c5f114d24a2965f6329f035ebf4b2d5b9ead87f37c568a3771b246"}
May 26 16:03:05.116802 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:05.116700    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6glcl" podStartSLOduration=131.239216048 podStartE2EDuration="2m12.116682611s" podCreationTimestamp="2026-05-26 16:00:53 +0000 UTC" firstStartedPulling="2026-05-26 16:03:03.656574635 +0000 UTC m=+130.528833047" lastFinishedPulling="2026-05-26 16:03:04.534041196 +0000 UTC m=+131.406299610" observedRunningTime="2026-05-26 16:03:05.115810321 +0000 UTC m=+131.988068756" watchObservedRunningTime="2026-05-26 16:03:05.116682611 +0000 UTC m=+131.988941045"
May 26 16:03:09.254817 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.254775    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"]
May 26 16:03:09.257274 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.257252    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"
May 26 16:03:09.261146 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.261123    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ngvcm\""
May 26 16:03:09.261146 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.261133    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
May 26 16:03:09.261146 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.261143    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
May 26 16:03:09.270007 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.269979    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"]
May 26 16:03:09.337880 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.337850    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7829b"]
May 26 16:03:09.340526 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.340509    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.343906 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.343865    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 26 16:03:09.344121 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.343931    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 26 16:03:09.344121 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.343935    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 26 16:03:09.344332 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.343941    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q4tcs\""
May 26 16:03:09.344332 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.343966    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 26 16:03:09.349606 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.349583    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5c76476c85-rff5w"]
May 26 16:03:09.351648 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.351633    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.351884 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.351859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf1ccb53-7cd2-4015-bc33-36ac74bc22b5-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-78ktw\" (UID: \"bf1ccb53-7cd2-4015-bc33-36ac74bc22b5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"
May 26 16:03:09.352049 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.351990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf1ccb53-7cd2-4015-bc33-36ac74bc22b5-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-78ktw\" (UID: \"bf1ccb53-7cd2-4015-bc33-36ac74bc22b5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"
May 26 16:03:09.354206 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.354187    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
May 26 16:03:09.354302 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.354208    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-n6l5h\""
May 26 16:03:09.354493 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.354480    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
May 26 16:03:09.354648 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.354630    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
May 26 16:03:09.360807 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.360785    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
May 26 16:03:09.366520 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.366497    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7829b"]
May 26 16:03:09.370107 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.370060    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c76476c85-rff5w"]
May 26 16:03:09.453139 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453105    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-ca-trust-extracted\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.453139 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453139    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-registry-certificates\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.453349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453204    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-trusted-ca\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.453349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453247    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-image-registry-private-configuration\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.453349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453285    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf1ccb53-7cd2-4015-bc33-36ac74bc22b5-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-78ktw\" (UID: \"bf1ccb53-7cd2-4015-bc33-36ac74bc22b5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"
May 26 16:03:09.453349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453311    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03a4348a-f490-4fba-94b1-3e0e1d82c14b-crio-socket\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.453349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03a4348a-f490-4fba-94b1-3e0e1d82c14b-data-volume\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.453349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453350    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a4348a-f490-4fba-94b1-3e0e1d82c14b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.453561 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453373    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf1ccb53-7cd2-4015-bc33-36ac74bc22b5-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-78ktw\" (UID: \"bf1ccb53-7cd2-4015-bc33-36ac74bc22b5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"
May 26 16:03:09.453561 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a4348a-f490-4fba-94b1-3e0e1d82c14b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.453561 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453517    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-installation-pull-secrets\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.453680 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453579    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk8sz\" (UniqueName: \"kubernetes.io/projected/03a4348a-f490-4fba-94b1-3e0e1d82c14b-kube-api-access-xk8sz\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.453680 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453619    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-bound-sa-token\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.453680 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453649    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7g6v\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-kube-api-access-n7g6v\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.453787 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.453688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-registry-tls\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.454243 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.454219    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bf1ccb53-7cd2-4015-bc33-36ac74bc22b5-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-78ktw\" (UID: \"bf1ccb53-7cd2-4015-bc33-36ac74bc22b5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"
May 26 16:03:09.455938 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.455875    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf1ccb53-7cd2-4015-bc33-36ac74bc22b5-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-78ktw\" (UID: \"bf1ccb53-7cd2-4015-bc33-36ac74bc22b5\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"
May 26 16:03:09.554992 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.554963    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-installation-pull-secrets\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555178 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.554998    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xk8sz\" (UniqueName: \"kubernetes.io/projected/03a4348a-f490-4fba-94b1-3e0e1d82c14b-kube-api-access-xk8sz\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.555178 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555017    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-bound-sa-token\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555295 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555236    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-n7g6v\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-kube-api-access-n7g6v\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555295 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555274    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-registry-tls\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555295 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555291    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-ca-trust-extracted\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555442 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555408    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-registry-certificates\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555464    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-trusted-ca\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555545 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555506    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-image-registry-private-configuration\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555545 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555541    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03a4348a-f490-4fba-94b1-3e0e1d82c14b-crio-socket\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.555641 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555568    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03a4348a-f490-4fba-94b1-3e0e1d82c14b-data-volume\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.555641 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555611    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a4348a-f490-4fba-94b1-3e0e1d82c14b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.555641 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555623    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/03a4348a-f490-4fba-94b1-3e0e1d82c14b-crio-socket\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.555792 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555642    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-ca-trust-extracted\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.555792 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555647    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a4348a-f490-4fba-94b1-3e0e1d82c14b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.555984 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.555963    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/03a4348a-f490-4fba-94b1-3e0e1d82c14b-data-volume\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.556487 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.556276    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/03a4348a-f490-4fba-94b1-3e0e1d82c14b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.556611 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.556337    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-registry-certificates\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.556828 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.556805    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-trusted-ca\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.557779 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.557755    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-installation-pull-secrets\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.557992 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.557975    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/03a4348a-f490-4fba-94b1-3e0e1d82c14b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.558218 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.558197    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-registry-tls\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.558882 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.558863    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-image-registry-private-configuration\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.565577 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.565558    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"
May 26 16:03:09.567030 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.567006    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk8sz\" (UniqueName: \"kubernetes.io/projected/03a4348a-f490-4fba-94b1-3e0e1d82c14b-kube-api-access-xk8sz\") pod \"insights-runtime-extractor-7829b\" (UID: \"03a4348a-f490-4fba-94b1-3e0e1d82c14b\") " pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.567755 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.567736    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-bound-sa-token\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.568433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.568407    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7g6v\" (UniqueName: \"kubernetes.io/projected/3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c-kube-api-access-n7g6v\") pod \"image-registry-5c76476c85-rff5w\" (UID: \"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c\") " pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.651268 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.651234    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7829b"
May 26 16:03:09.662842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.662821    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:09.697708 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.697650    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw"]
May 26 16:03:09.704043 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:09.703974    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1ccb53_7cd2_4015_bc33_36ac74bc22b5.slice/crio-5cb84c30c2380d9440f53b33553d5509e12eb7f7d36f5ccd5010b3aeae5c3166 WatchSource:0}: Error finding container 5cb84c30c2380d9440f53b33553d5509e12eb7f7d36f5ccd5010b3aeae5c3166: Status 404 returned error can't find the container with id 5cb84c30c2380d9440f53b33553d5509e12eb7f7d36f5ccd5010b3aeae5c3166
May 26 16:03:09.810240 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.810149    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7829b"]
May 26 16:03:09.813108 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:09.813076    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a4348a_f490_4fba_94b1_3e0e1d82c14b.slice/crio-a419aec3c60b795e4ec4dc2bf9fe305f555cf1012078f2537c01091cf7238fe7 WatchSource:0}: Error finding container a419aec3c60b795e4ec4dc2bf9fe305f555cf1012078f2537c01091cf7238fe7: Status 404 returned error can't find the container with id a419aec3c60b795e4ec4dc2bf9fe305f555cf1012078f2537c01091cf7238fe7
May 26 16:03:09.844540 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:09.844463    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5c76476c85-rff5w"]
May 26 16:03:09.847176 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:09.847148    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f71ece4_23ab_41d8_adcf_89f3bdc1ec5c.slice/crio-163a61647a4def2498f4ea9ce47a0573776c53a4f18708542506ee80f132705f WatchSource:0}: Error finding container 163a61647a4def2498f4ea9ce47a0573776c53a4f18708542506ee80f132705f: Status 404 returned error can't find the container with id 163a61647a4def2498f4ea9ce47a0573776c53a4f18708542506ee80f132705f
May 26 16:03:10.112834 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:10.112755    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw" event={"ID":"bf1ccb53-7cd2-4015-bc33-36ac74bc22b5","Type":"ContainerStarted","Data":"5cb84c30c2380d9440f53b33553d5509e12eb7f7d36f5ccd5010b3aeae5c3166"}
May 26 16:03:10.114073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:10.114046    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7829b" event={"ID":"03a4348a-f490-4fba-94b1-3e0e1d82c14b","Type":"ContainerStarted","Data":"2935e25db9c7d50880466fbe12dfbee00895ec766353cdcadd8bd6f5d8c0e613"}
May 26 16:03:10.114194 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:10.114082    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7829b" event={"ID":"03a4348a-f490-4fba-94b1-3e0e1d82c14b","Type":"ContainerStarted","Data":"a419aec3c60b795e4ec4dc2bf9fe305f555cf1012078f2537c01091cf7238fe7"}
May 26 16:03:10.115262 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:10.115237    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" event={"ID":"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c","Type":"ContainerStarted","Data":"c8c39f22283e1f6bac23ff5ae914e40f878d6302eb9454ad39b8bf09a1f6f0bb"}
May 26 16:03:10.115348 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:10.115266    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" event={"ID":"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c","Type":"ContainerStarted","Data":"163a61647a4def2498f4ea9ce47a0573776c53a4f18708542506ee80f132705f"}
May 26 16:03:10.115402 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:10.115392    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:10.143013 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:10.142974    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" podStartSLOduration=1.142959868 podStartE2EDuration="1.142959868s" podCreationTimestamp="2026-05-26 16:03:09 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:03:10.141975439 +0000 UTC m=+137.014233872" watchObservedRunningTime="2026-05-26 16:03:10.142959868 +0000 UTC m=+137.015218302"
May 26 16:03:11.119094 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:11.119053    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw" event={"ID":"bf1ccb53-7cd2-4015-bc33-36ac74bc22b5","Type":"ContainerStarted","Data":"b5846658de752ba34c2d0834defd1f09cd7274ea557eba22eb93b6f53606f709"}
May 26 16:03:11.120653 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:11.120621    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7829b" event={"ID":"03a4348a-f490-4fba-94b1-3e0e1d82c14b","Type":"ContainerStarted","Data":"509b3f53ca8c01dc4d6234ed4328d26e9d4986ac8b192acc3126585691460adc"}
May 26 16:03:11.136823 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:11.136769    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-78ktw" podStartSLOduration=0.901606874 podStartE2EDuration="2.136753316s" podCreationTimestamp="2026-05-26 16:03:09 +0000 UTC" firstStartedPulling="2026-05-26 16:03:09.706620983 +0000 UTC m=+136.578879415" lastFinishedPulling="2026-05-26 16:03:10.941767446 +0000 UTC m=+137.814025857" observedRunningTime="2026-05-26 16:03:11.135206983 +0000 UTC m=+138.007465440" watchObservedRunningTime="2026-05-26 16:03:11.136753316 +0000 UTC m=+138.009011751"
May 26 16:03:13.087735 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.087701    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c7d67fc8-dpwmn"]
May 26 16:03:13.089734 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.089715    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.092694 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.092251    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ssfl9\""
May 26 16:03:13.092694 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.092601    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 26 16:03:13.092866 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.092854    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 26 16:03:13.094073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.093326    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 26 16:03:13.094073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.093435    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 26 16:03:13.094073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.093583    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 26 16:03:13.094073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.093334    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 26 16:03:13.094073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.093825    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 26 16:03:13.099914 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.099876    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c7d67fc8-dpwmn"]
May 26 16:03:13.127617 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.127583    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7829b" event={"ID":"03a4348a-f490-4fba-94b1-3e0e1d82c14b","Type":"ContainerStarted","Data":"0a68e5601c1d5664fb51f0561722bb466c0efff2bfb5c20bdf4d5435513dfd19"}
May 26 16:03:13.148619 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.148576    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7829b" podStartSLOduration=1.829899076 podStartE2EDuration="4.148564015s" podCreationTimestamp="2026-05-26 16:03:09 +0000 UTC" firstStartedPulling="2026-05-26 16:03:09.869986102 +0000 UTC m=+136.742244514" lastFinishedPulling="2026-05-26 16:03:12.188651035 +0000 UTC m=+139.060909453" observedRunningTime="2026-05-26 16:03:13.147538375 +0000 UTC m=+140.019796809" watchObservedRunningTime="2026-05-26 16:03:13.148564015 +0000 UTC m=+140.020822449"
May 26 16:03:13.188197 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.188169    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-oauth-config\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.188309 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.188218    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4w62\" (UniqueName: \"kubernetes.io/projected/f84bc018-3054-4e53-aee0-e146219cb9bb-kube-api-access-r4w62\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.188309 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.188238    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-console-config\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.188309 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.188294    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-serving-cert\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.188443 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.188321    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-service-ca\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.188443 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.188354    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-oauth-serving-cert\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.289625 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.289585    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-oauth-serving-cert\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.289792 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.289769    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-oauth-config\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.289885 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.289867    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r4w62\" (UniqueName: \"kubernetes.io/projected/f84bc018-3054-4e53-aee0-e146219cb9bb-kube-api-access-r4w62\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.289970 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.289930    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-console-config\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.290044 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.290003    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-serving-cert\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.290044 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.290032    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-service-ca\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.290597 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.290375    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-oauth-serving-cert\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.290723 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.290617    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-console-config\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.290723 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.290667    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-service-ca\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.292162 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.292139    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-oauth-config\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.292425 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.292407    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-serving-cert\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.298195 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.298176    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4w62\" (UniqueName: \"kubernetes.io/projected/f84bc018-3054-4e53-aee0-e146219cb9bb-kube-api-access-r4w62\") pod \"console-85c7d67fc8-dpwmn\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") " pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.401331 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.401251    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:13.525755 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:13.525726    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c7d67fc8-dpwmn"]
May 26 16:03:13.527925 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:13.527881    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84bc018_3054_4e53_aee0_e146219cb9bb.slice/crio-887e1afe9d48366b57839c082327939c97e271ea5af658e162c2776e51cf8b37 WatchSource:0}: Error finding container 887e1afe9d48366b57839c082327939c97e271ea5af658e162c2776e51cf8b37: Status 404 returned error can't find the container with id 887e1afe9d48366b57839c082327939c97e271ea5af658e162c2776e51cf8b37
May 26 16:03:14.131743 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:14.131700    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c7d67fc8-dpwmn" event={"ID":"f84bc018-3054-4e53-aee0-e146219cb9bb","Type":"ContainerStarted","Data":"887e1afe9d48366b57839c082327939c97e271ea5af658e162c2776e51cf8b37"}
May 26 16:03:17.144141 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.144106    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c7d67fc8-dpwmn" event={"ID":"f84bc018-3054-4e53-aee0-e146219cb9bb","Type":"ContainerStarted","Data":"0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2"}
May 26 16:03:17.163632 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.163576    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-85c7d67fc8-dpwmn" podStartSLOduration=1.495865356 podStartE2EDuration="4.163561137s" podCreationTimestamp="2026-05-26 16:03:13 +0000 UTC" firstStartedPulling="2026-05-26 16:03:13.529860799 +0000 UTC m=+140.402119211" lastFinishedPulling="2026-05-26 16:03:16.197556577 +0000 UTC m=+143.069814992" observedRunningTime="2026-05-26 16:03:17.162215549 +0000 UTC m=+144.034473983" watchObservedRunningTime="2026-05-26 16:03:17.163561137 +0000 UTC m=+144.035819570"
May 26 16:03:17.803618 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.803578    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"]
May 26 16:03:17.806245 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.806221    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:17.809154 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.809131    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 26 16:03:17.810001 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.809980    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qpsdj\""
May 26 16:03:17.810112 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.810012    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 26 16:03:17.810269 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.810254    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 26 16:03:17.810548 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.810494    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 26 16:03:17.822634 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.822610    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qqf57"]
May 26 16:03:17.825251 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.825231    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.825929 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.825638    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"]
May 26 16:03:17.828763 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.828443    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cb4pg\""
May 26 16:03:17.828763 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.828520    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 26 16:03:17.828763 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.828583    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 26 16:03:17.828763 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.828659    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 26 16:03:17.929544 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-sys\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.929712 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:17.929712 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929587    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-root\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.929712 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929618    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084ed376-fc35-4e25-ac53-404b0d311e87-metrics-client-ca\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.929924 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mf9\" (UniqueName: \"kubernetes.io/projected/084ed376-fc35-4e25-ac53-404b0d311e87-kube-api-access-t5mf9\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.929924 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-textfile\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.929924 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929771    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.929924 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929810    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:17.929924 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:17.929924 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929878    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:17.929924 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-wtmp\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.930255 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929949    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:17.930255 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.929991    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:17.930255 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.930015    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mbp\" (UniqueName: \"kubernetes.io/projected/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-api-access-m6mbp\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:17.930255 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:17.930038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-tls\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.030745 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.030702    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-sys\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.030972 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.030763    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.030972 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.030800    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-root\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.030972 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.030832    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084ed376-fc35-4e25-ac53-404b0d311e87-metrics-client-ca\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.030972 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.030872    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mf9\" (UniqueName: \"kubernetes.io/projected/084ed376-fc35-4e25-ac53-404b0d311e87-kube-api-access-t5mf9\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.030972 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.030919    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-textfile\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.030972 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.030947    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.031259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.030983    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.031259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.031019    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.031259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.031053    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.031259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.031079    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-wtmp\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.031259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.031109    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.031259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.031147    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.031259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.031174    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mbp\" (UniqueName: \"kubernetes.io/projected/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-api-access-m6mbp\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.031259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.031201    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-tls\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.031636 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:18.031341    2561 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
May 26 16:03:18.031636 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:18.031430    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-tls podName:084ed376-fc35-4e25-ac53-404b0d311e87 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:18.531410757 +0000 UTC m=+145.403669175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-tls") pod "node-exporter-qqf57" (UID: "084ed376-fc35-4e25-ac53-404b0d311e87") : secret "node-exporter-tls" not found
May 26 16:03:18.031743 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.031703    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-sys\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.032974 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:18.032013    2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 26 16:03:18.032974 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:18.032088    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls podName:74a23d36-6c2e-405e-83d9-9b6ca1ac9185 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:18.532070448 +0000 UTC m=+145.404328868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-sq547" (UID: "74a23d36-6c2e-405e-83d9-9b6ca1ac9185") : secret "kube-state-metrics-tls" not found
May 26 16:03:18.032974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.032156    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-root\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.032974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.032396    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-textfile\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.032974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.032690    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/084ed376-fc35-4e25-ac53-404b0d311e87-metrics-client-ca\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.032974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.032812    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.032974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.032910    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-accelerators-collector-config\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.033402 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.033062    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-wtmp\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.033402 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.033157    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.033652 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.033602    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.035792 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.035771    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.036182 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.036160    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.041128 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.041101    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mf9\" (UniqueName: \"kubernetes.io/projected/084ed376-fc35-4e25-ac53-404b0d311e87-kube-api-access-t5mf9\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.043320 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.043297    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mbp\" (UniqueName: \"kubernetes.io/projected/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-api-access-m6mbp\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.535680 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.535632    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:18.536214 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.535703    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-tls\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.536214 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:18.535832    2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
May 26 16:03:18.536214 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:18.535970    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls podName:74a23d36-6c2e-405e-83d9-9b6ca1ac9185 nodeName:}" failed. No retries permitted until 2026-05-26 16:03:19.535946932 +0000 UTC m=+146.408205350 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-sq547" (UID: "74a23d36-6c2e-405e-83d9-9b6ca1ac9185") : secret "kube-state-metrics-tls" not found
May 26 16:03:18.538056 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.538035    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/084ed376-fc35-4e25-ac53-404b0d311e87-node-exporter-tls\") pod \"node-exporter-qqf57\" (UID: \"084ed376-fc35-4e25-ac53-404b0d311e87\") " pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.736123 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.736090    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qqf57"
May 26 16:03:18.745822 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:18.745787    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084ed376_fc35_4e25_ac53_404b0d311e87.slice/crio-cfe75c943261163374364c7c0f59830fb8ba8bd93b4f41abf5e63f067860009e WatchSource:0}: Error finding container cfe75c943261163374364c7c0f59830fb8ba8bd93b4f41abf5e63f067860009e: Status 404 returned error can't find the container with id cfe75c943261163374364c7c0f59830fb8ba8bd93b4f41abf5e63f067860009e
May 26 16:03:18.875214 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.875136    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:03:18.889938 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.889884    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:18.894751 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.894703    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 26 16:03:18.894751 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.894722    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 26 16:03:18.896341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.895404    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 26 16:03:18.896341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.895496    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 26 16:03:18.896341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.895503    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 26 16:03:18.896341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.895866    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-gdncm\""
May 26 16:03:18.896341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.896006    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 26 16:03:18.896341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.896120    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 26 16:03:18.896341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.896189    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 26 16:03:18.896341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.896286    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 26 16:03:18.897022 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:18.896709    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:03:19.041236 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041192    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041452 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041264    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-web-config\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041452 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041290    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041452 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041310    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041452 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041598 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041598 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlpm\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-kube-api-access-tqlpm\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041598 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041557    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041598 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041582    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041756 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041756 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041756 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041713    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-out\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.041756 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.041738    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143048 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.142978    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143045    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143074    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlpm\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-kube-api-access-tqlpm\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143092    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143109    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143125    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143155    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143176    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-out\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143197    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143246 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143242    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143702 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143479    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-web-config\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143702 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143520    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.143702 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.143548    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.146205 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.145037    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.146205 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.145879    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.146381 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.146247    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.146952 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.146735    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.147237 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.147210    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.148054 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.147951    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.149313 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.148728    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.149313 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.148755    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-web-config\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.149313 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.149187    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.149529 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.149408    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.149585 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.149546    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-out\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.150455 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.150433    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.151185 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.151156    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqf57" event={"ID":"084ed376-fc35-4e25-ac53-404b0d311e87","Type":"ContainerStarted","Data":"cfe75c943261163374364c7c0f59830fb8ba8bd93b4f41abf5e63f067860009e"}
May 26 16:03:19.153513 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.153492    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlpm\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-kube-api-access-tqlpm\") pod \"alertmanager-main-0\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") " pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.202261 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.202206    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:03:19.355519 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.355446    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:03:19.359668 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:19.359607    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3325d4c_ac99_44b4_b576_eae4b16b3830.slice/crio-b66db5d891798dc37b83e5263b7e469f4255e3e32185f6b4baf032f0135de6f4 WatchSource:0}: Error finding container b66db5d891798dc37b83e5263b7e469f4255e3e32185f6b4baf032f0135de6f4: Status 404 returned error can't find the container with id b66db5d891798dc37b83e5263b7e469f4255e3e32185f6b4baf032f0135de6f4
May 26 16:03:19.547111 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.547077    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:19.549773 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.549746    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/74a23d36-6c2e-405e-83d9-9b6ca1ac9185-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-sq547\" (UID: \"74a23d36-6c2e-405e-83d9-9b6ca1ac9185\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:19.616772 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.616752    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"
May 26 16:03:19.745757 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:19.745550    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547"]
May 26 16:03:19.748371 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:19.748329    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a23d36_6c2e_405e_83d9_9b6ca1ac9185.slice/crio-2bb69a3a09d2dc4d80a74b52dc022a87b418cc00e1bbe7a7004f71e985f686d3 WatchSource:0}: Error finding container 2bb69a3a09d2dc4d80a74b52dc022a87b418cc00e1bbe7a7004f71e985f686d3: Status 404 returned error can't find the container with id 2bb69a3a09d2dc4d80a74b52dc022a87b418cc00e1bbe7a7004f71e985f686d3
May 26 16:03:20.155402 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:20.155363    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547" event={"ID":"74a23d36-6c2e-405e-83d9-9b6ca1ac9185","Type":"ContainerStarted","Data":"2bb69a3a09d2dc4d80a74b52dc022a87b418cc00e1bbe7a7004f71e985f686d3"}
May 26 16:03:20.156488 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:20.156458    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerStarted","Data":"b66db5d891798dc37b83e5263b7e469f4255e3e32185f6b4baf032f0135de6f4"}
May 26 16:03:20.157784 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:20.157756    2561 generic.go:358] "Generic (PLEG): container finished" podID="084ed376-fc35-4e25-ac53-404b0d311e87" containerID="6b707e94a2699c0e7810416815294780d4879a30cba0d758d058abd3c19453cd" exitCode=0
May 26 16:03:20.157927 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:20.157793    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqf57" event={"ID":"084ed376-fc35-4e25-ac53-404b0d311e87","Type":"ContainerDied","Data":"6b707e94a2699c0e7810416815294780d4879a30cba0d758d058abd3c19453cd"}
May 26 16:03:21.162457 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:21.162429    2561 generic.go:358] "Generic (PLEG): container finished" podID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerID="0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188" exitCode=0
May 26 16:03:21.162811 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:21.162508    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerDied","Data":"0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188"}
May 26 16:03:21.164957 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:21.164926    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqf57" event={"ID":"084ed376-fc35-4e25-ac53-404b0d311e87","Type":"ContainerStarted","Data":"25eec6f1b999202244d3345c4839d5a70ea57a6bad6e15be0e3cc2aa4491a261"}
May 26 16:03:21.165073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:21.164967    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qqf57" event={"ID":"084ed376-fc35-4e25-ac53-404b0d311e87","Type":"ContainerStarted","Data":"142ef482f200909bd5dfe1b0833553c8030e8ae3c5dc672c77aee8c3d43e34e4"}
May 26 16:03:21.166340 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:21.166321    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547" event={"ID":"74a23d36-6c2e-405e-83d9-9b6ca1ac9185","Type":"ContainerStarted","Data":"ec9421af727592379236f5072707df8175a27b9b2bbb9a6f8774c75c0654ca5b"}
May 26 16:03:21.211595 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:21.210639    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qqf57" podStartSLOduration=3.34542926 podStartE2EDuration="4.210616255s" podCreationTimestamp="2026-05-26 16:03:17 +0000 UTC" firstStartedPulling="2026-05-26 16:03:18.747531743 +0000 UTC m=+145.619790159" lastFinishedPulling="2026-05-26 16:03:19.612718727 +0000 UTC m=+146.484977154" observedRunningTime="2026-05-26 16:03:21.209209808 +0000 UTC m=+148.081468237" watchObservedRunningTime="2026-05-26 16:03:21.210616255 +0000 UTC m=+148.082874688"
May 26 16:03:22.171020 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.170976    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547" event={"ID":"74a23d36-6c2e-405e-83d9-9b6ca1ac9185","Type":"ContainerStarted","Data":"ec249e91cebd94255c291e0df29d32a0609e47eddaf567992e4e0142a6962195"}
May 26 16:03:22.171020 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.171017    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547" event={"ID":"74a23d36-6c2e-405e-83d9-9b6ca1ac9185","Type":"ContainerStarted","Data":"bb160845483c92b9835ef7400994852186beb8c1648428b449ec73b1f08b483c"}
May 26 16:03:22.194092 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.194037    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-sq547" podStartSLOduration=3.873535117 podStartE2EDuration="5.194022984s" podCreationTimestamp="2026-05-26 16:03:17 +0000 UTC" firstStartedPulling="2026-05-26 16:03:19.750145102 +0000 UTC m=+146.622403515" lastFinishedPulling="2026-05-26 16:03:21.070632956 +0000 UTC m=+147.942891382" observedRunningTime="2026-05-26 16:03:22.192820064 +0000 UTC m=+149.065078498" watchObservedRunningTime="2026-05-26 16:03:22.194022984 +0000 UTC m=+149.066281418"
May 26 16:03:22.222499 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.222467    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-59874dc46b-hvndz"]
May 26 16:03:22.224674 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.224654    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.231096 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.231068    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4jzvw\""
May 26 16:03:22.231205 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.231068    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5g5l0jju0s973\""
May 26 16:03:22.233362 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.233342    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 26 16:03:22.233639 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.233622    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 26 16:03:22.233728 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.233704    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 26 16:03:22.233824 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.233806    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 26 16:03:22.244591 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.244566    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59874dc46b-hvndz"]
May 26 16:03:22.372261 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.372232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/20728c3e-d771-4696-9ab9-04a53ada173f-metrics-server-audit-profiles\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.372413 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.372275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-client-ca-bundle\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.372474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.372411    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/20728c3e-d771-4696-9ab9-04a53ada173f-audit-log\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.372474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.372464    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-secret-metrics-server-client-certs\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.372551 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.372493    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-secret-metrics-server-tls\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.372551 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.372512    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20728c3e-d771-4696-9ab9-04a53ada173f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.372551 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.372544    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6pd5\" (UniqueName: \"kubernetes.io/projected/20728c3e-d771-4696-9ab9-04a53ada173f-kube-api-access-f6pd5\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.473822 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.473727    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/20728c3e-d771-4696-9ab9-04a53ada173f-audit-log\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.473822 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.473772    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-secret-metrics-server-client-certs\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.473822 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.473798    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-secret-metrics-server-tls\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.474114 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.474015    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20728c3e-d771-4696-9ab9-04a53ada173f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.474114 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.474071    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f6pd5\" (UniqueName: \"kubernetes.io/projected/20728c3e-d771-4696-9ab9-04a53ada173f-kube-api-access-f6pd5\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.474439 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.474358    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/20728c3e-d771-4696-9ab9-04a53ada173f-metrics-server-audit-profiles\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.474439 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.474402    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/20728c3e-d771-4696-9ab9-04a53ada173f-audit-log\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.474439 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.474405    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-client-ca-bundle\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.474814 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.474793    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20728c3e-d771-4696-9ab9-04a53ada173f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.475373 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.475343    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/20728c3e-d771-4696-9ab9-04a53ada173f-metrics-server-audit-profiles\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.476986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.476942    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-secret-metrics-server-tls\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.477080 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.476997    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-secret-metrics-server-client-certs\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.477338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.477296    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/20728c3e-d771-4696-9ab9-04a53ada173f-client-ca-bundle\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.483335 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.483311    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6pd5\" (UniqueName: \"kubernetes.io/projected/20728c3e-d771-4696-9ab9-04a53ada173f-kube-api-access-f6pd5\") pod \"metrics-server-59874dc46b-hvndz\" (UID: \"20728c3e-d771-4696-9ab9-04a53ada173f\") " pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.535556 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.535452    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:22.840371 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:22.840344    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-59874dc46b-hvndz"]
May 26 16:03:22.843809 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:22.843781    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20728c3e_d771_4696_9ab9_04a53ada173f.slice/crio-857bbcd2c9aa9fd09f1a27012df001c38a18b902b63aa3ec9b6427ef16892ee6 WatchSource:0}: Error finding container 857bbcd2c9aa9fd09f1a27012df001c38a18b902b63aa3ec9b6427ef16892ee6: Status 404 returned error can't find the container with id 857bbcd2c9aa9fd09f1a27012df001c38a18b902b63aa3ec9b6427ef16892ee6
May 26 16:03:23.175957 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.175921    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerStarted","Data":"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80"}
May 26 16:03:23.175957 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.175961    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerStarted","Data":"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765"}
May 26 16:03:23.176431 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.175976    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerStarted","Data":"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2"}
May 26 16:03:23.176431 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.175990    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerStarted","Data":"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6"}
May 26 16:03:23.176431 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.176005    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerStarted","Data":"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7"}
May 26 16:03:23.176980 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.176957    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz" event={"ID":"20728c3e-d771-4696-9ab9-04a53ada173f","Type":"ContainerStarted","Data":"857bbcd2c9aa9fd09f1a27012df001c38a18b902b63aa3ec9b6427ef16892ee6"}
May 26 16:03:23.402453 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.402406    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:23.402453 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.402457    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:23.404068 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.404036    2561 patch_prober.go:28] interesting pod/console-85c7d67fc8-dpwmn container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused" start-of-body=
May 26 16:03:23.404229 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:23.404136    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-85c7d67fc8-dpwmn" podUID="f84bc018-3054-4e53-aee0-e146219cb9bb" containerName="console" probeResult="failure" output="Get \"https://10.134.0.15:8443/health\": dial tcp 10.134.0.15:8443: connect: connection refused"
May 26 16:03:24.071722 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.071619    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:03:24.075361 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.075338    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.080231 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.080207    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 26 16:03:24.080365 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.080270    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 26 16:03:24.080365 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.080325    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 26 16:03:24.081110 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.081080    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 26 16:03:24.082917 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.082744    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 26 16:03:24.083587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.083000    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5bl6nro5d0n34\""
May 26 16:03:24.083587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.083093    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 26 16:03:24.083587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.083208    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-knhsq\""
May 26 16:03:24.083587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.083377    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 26 16:03:24.083587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.083390    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 26 16:03:24.083587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.083503    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 26 16:03:24.084022 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.083749    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 26 16:03:24.086563 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.086671 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086576    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.086671 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.086770 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086678    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.086770 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwnbx\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-kube-api-access-fwnbx\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.086770 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.086937 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086856    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.086937 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086918    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087044 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086953    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087044 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.086983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087044 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087009    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087183 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-config\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087183 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087101    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087183 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087131    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087183 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087156    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087376 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087191    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087376 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087225    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087376 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.087806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.087788    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 26 16:03:24.090811 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.090788    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 26 16:03:24.094770 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.094747    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:03:24.184541 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.184499    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerStarted","Data":"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985"}
May 26 16:03:24.187685 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.187630    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.187685 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.187660    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.187723    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.187958    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188031    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fwnbx\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-kube-api-access-fwnbx\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188071    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188180    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188215    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188263    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188295    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188326    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188367    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-config\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188393    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188435    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188470    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188537    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188581    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.188639    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.190565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.189224    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.191565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.189374    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.191565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.190015    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.192288 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.192261    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.193087 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.192700    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-web-config\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.193645 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.193465    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.194016 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.193992    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.195078 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.195054    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.196099 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.195957    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.198088 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.196395    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.198673 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.198618    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.199274 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.199219    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.201971 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.201831    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.202208 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.201957    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-config\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.203573 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.203498    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.203844 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.203827    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-config-out\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.206269 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.206212    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.206966 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.206937    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwnbx\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-kube-api-access-fwnbx\") pod \"prometheus-k8s-0\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.229055 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.228994    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.7733095300000001 podStartE2EDuration="6.228978654s" podCreationTimestamp="2026-05-26 16:03:18 +0000 UTC" firstStartedPulling="2026-05-26 16:03:19.364053068 +0000 UTC m=+146.236311485" lastFinishedPulling="2026-05-26 16:03:23.819722185 +0000 UTC m=+150.691980609" observedRunningTime="2026-05-26 16:03:24.226922446 +0000 UTC m=+151.099180880" watchObservedRunningTime="2026-05-26 16:03:24.228978654 +0000 UTC m=+151.101237088"
May 26 16:03:24.387155 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.387116    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:24.541586 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:24.541544    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:03:24.545153 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:24.545125    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c2732de_3d29_4700_bf6c_71f1522ef58d.slice/crio-72f08609a52554590447fe7af270ba9ba53bef09d70bfa34aac7fed700f6c67b WatchSource:0}: Error finding container 72f08609a52554590447fe7af270ba9ba53bef09d70bfa34aac7fed700f6c67b: Status 404 returned error can't find the container with id 72f08609a52554590447fe7af270ba9ba53bef09d70bfa34aac7fed700f6c67b
May 26 16:03:25.190919 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:25.190869    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz" event={"ID":"20728c3e-d771-4696-9ab9-04a53ada173f","Type":"ContainerStarted","Data":"2232896b17f94006e1c03d2b0bf9312870d920359b2f6cec529dac487e15f464"}
May 26 16:03:25.192230 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:25.192204    2561 generic.go:358] "Generic (PLEG): container finished" podID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerID="037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d" exitCode=0
May 26 16:03:25.192344 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:25.192287    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerDied","Data":"037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d"}
May 26 16:03:25.192344 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:25.192319    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerStarted","Data":"72f08609a52554590447fe7af270ba9ba53bef09d70bfa34aac7fed700f6c67b"}
May 26 16:03:25.209241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:25.209198    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz" podStartSLOduration=1.6639941999999999 podStartE2EDuration="3.209182394s" podCreationTimestamp="2026-05-26 16:03:22 +0000 UTC" firstStartedPulling="2026-05-26 16:03:22.846061074 +0000 UTC m=+149.718319504" lastFinishedPulling="2026-05-26 16:03:24.391249286 +0000 UTC m=+151.263507698" observedRunningTime="2026-05-26 16:03:25.208165724 +0000 UTC m=+152.080424157" watchObservedRunningTime="2026-05-26 16:03:25.209182394 +0000 UTC m=+152.081440828"
May 26 16:03:25.491771 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:25.491741    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c7d67fc8-dpwmn"]
May 26 16:03:28.206160 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:28.206069    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerStarted","Data":"c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546"}
May 26 16:03:28.206160 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:28.206106    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerStarted","Data":"0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf"}
May 26 16:03:29.669076 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:29.669033    2561 patch_prober.go:28] interesting pod/image-registry-5c76476c85-rff5w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:03:29.669517 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:29.669096    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" podUID="3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:03:30.078884 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:30.078842    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-4qxkt" podUID="3b9eef08-2c0d-419e-b42e-5ad0cff81e71"
May 26 16:03:30.091272 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:30.091247    2561 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-p4xt6" podUID="78c7e49d-53fc-4af3-874c-8a7b6ab588e7"
May 26 16:03:30.215291 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:30.215214    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:03:30.215438 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:30.215215    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerStarted","Data":"784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6"}
May 26 16:03:30.215438 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:30.215319    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerStarted","Data":"be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42"}
May 26 16:03:30.215438 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:30.215328    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerStarted","Data":"8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47"}
May 26 16:03:30.215438 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:30.215336    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerStarted","Data":"41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97"}
May 26 16:03:30.259902 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:30.259835    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.654833446 podStartE2EDuration="6.259816739s" podCreationTimestamp="2026-05-26 16:03:24 +0000 UTC" firstStartedPulling="2026-05-26 16:03:25.193345297 +0000 UTC m=+152.065603709" lastFinishedPulling="2026-05-26 16:03:29.798328585 +0000 UTC m=+156.670587002" observedRunningTime="2026-05-26 16:03:30.256384404 +0000 UTC m=+157.128642849" watchObservedRunningTime="2026-05-26 16:03:30.259816739 +0000 UTC m=+157.132075174"
May 26 16:03:31.126542 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:31.126502    2561 patch_prober.go:28] interesting pod/image-registry-5c76476c85-rff5w container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:03:31.127046 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:31.126968    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" podUID="3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:03:34.387875 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:34.387822    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:03:34.895186 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:34.895145    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:03:34.897606 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:34.897579    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b9eef08-2c0d-419e-b42e-5ad0cff81e71-cert\") pod \"ingress-canary-4qxkt\" (UID: \"3b9eef08-2c0d-419e-b42e-5ad0cff81e71\") " pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:03:34.995941 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:34.995885    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:03:34.998237 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:34.998207    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/78c7e49d-53fc-4af3-874c-8a7b6ab588e7-metrics-tls\") pod \"dns-default-p4xt6\" (UID: \"78c7e49d-53fc-4af3-874c-8a7b6ab588e7\") " pod="openshift-dns/dns-default-p4xt6"
May 26 16:03:35.019135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:35.019108    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l7p5g\""
May 26 16:03:35.027421 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:35.027397    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4qxkt"
May 26 16:03:35.173179 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:35.173153    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4qxkt"]
May 26 16:03:35.175936 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:35.175902    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9eef08_2c0d_419e_b42e_5ad0cff81e71.slice/crio-5aa7841118f8bf9b4e0188acb4c9fce0f37747fe347f3a921a24bced49e20186 WatchSource:0}: Error finding container 5aa7841118f8bf9b4e0188acb4c9fce0f37747fe347f3a921a24bced49e20186: Status 404 returned error can't find the container with id 5aa7841118f8bf9b4e0188acb4c9fce0f37747fe347f3a921a24bced49e20186
May 26 16:03:35.232508 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:35.232475    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4qxkt" event={"ID":"3b9eef08-2c0d-419e-b42e-5ad0cff81e71","Type":"ContainerStarted","Data":"5aa7841118f8bf9b4e0188acb4c9fce0f37747fe347f3a921a24bced49e20186"}
May 26 16:03:37.241785 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:37.241748    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4qxkt" event={"ID":"3b9eef08-2c0d-419e-b42e-5ad0cff81e71","Type":"ContainerStarted","Data":"bd6c177267b1d4c8545696d522d9370dfb5545c9ea228ad629d0f5c8a3fbb41a"}
May 26 16:03:37.262099 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:37.262058    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4qxkt" podStartSLOduration=128.699352812 podStartE2EDuration="2m10.262045314s" podCreationTimestamp="2026-05-26 16:01:27 +0000 UTC" firstStartedPulling="2026-05-26 16:03:35.178027486 +0000 UTC m=+162.050285901" lastFinishedPulling="2026-05-26 16:03:36.740719991 +0000 UTC m=+163.612978403" observedRunningTime="2026-05-26 16:03:37.259932392 +0000 UTC m=+164.132190826" watchObservedRunningTime="2026-05-26 16:03:37.262045314 +0000 UTC m=+164.134303748"
May 26 16:03:38.765394 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.765360    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dd78598cd-xk7tp"]
May 26 16:03:38.768506 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.768490    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.776748 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.776726    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 26 16:03:38.781273 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.781250    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd78598cd-xk7tp"]
May 26 16:03:38.830173 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.830146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-oauth-serving-cert\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.830173 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.830178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-oauth-config\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.830411 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.830216    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-trusted-ca-bundle\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.830411 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.830312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-config\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.830411 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.830348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-service-ca\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.830561 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.830442    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-serving-cert\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.830561 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.830474    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znvbm\" (UniqueName: \"kubernetes.io/projected/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-kube-api-access-znvbm\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.931869 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.931782    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-serving-cert\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.931869 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.931829    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-znvbm\" (UniqueName: \"kubernetes.io/projected/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-kube-api-access-znvbm\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.931869 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.931873    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-oauth-serving-cert\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.932216 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.931926    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-oauth-config\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.932216 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.931964    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-trusted-ca-bundle\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.932216 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.932013    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-config\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.932216 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.932033    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-service-ca\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.932673 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.932649    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-oauth-serving-cert\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.932776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.932761    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-service-ca\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.932821 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.932761    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-config\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.932967 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.932943    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-trusted-ca-bundle\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.934264 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.934240    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-oauth-config\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.934558 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.934538    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-serving-cert\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:38.941130 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:38.941106    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znvbm\" (UniqueName: \"kubernetes.io/projected/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-kube-api-access-znvbm\") pod \"console-7dd78598cd-xk7tp\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") " pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:39.077508 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:39.077425    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:39.204671 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:39.204577    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dd78598cd-xk7tp"]
May 26 16:03:39.207198 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:39.207169    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477d8ab3_c4c8_4ca9_83d5_0fc8c772ec53.slice/crio-cffd5a5e50a451f2538377edefe5a2cb7b8ebc38b0793ff8f08d211464f1ec79 WatchSource:0}: Error finding container cffd5a5e50a451f2538377edefe5a2cb7b8ebc38b0793ff8f08d211464f1ec79: Status 404 returned error can't find the container with id cffd5a5e50a451f2538377edefe5a2cb7b8ebc38b0793ff8f08d211464f1ec79
May 26 16:03:39.249130 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:39.249088    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd78598cd-xk7tp" event={"ID":"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53","Type":"ContainerStarted","Data":"cffd5a5e50a451f2538377edefe5a2cb7b8ebc38b0793ff8f08d211464f1ec79"}
May 26 16:03:39.667173 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:39.667143    2561 patch_prober.go:28] interesting pod/image-registry-5c76476c85-rff5w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:03:39.667338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:39.667200    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" podUID="3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:03:40.253834 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:40.253793    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd78598cd-xk7tp" event={"ID":"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53","Type":"ContainerStarted","Data":"ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f"}
May 26 16:03:40.272754 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:40.272705    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7dd78598cd-xk7tp" podStartSLOduration=2.272688051 podStartE2EDuration="2.272688051s" podCreationTimestamp="2026-05-26 16:03:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:03:40.271475643 +0000 UTC m=+167.143734074" watchObservedRunningTime="2026-05-26 16:03:40.272688051 +0000 UTC m=+167.144946479"
May 26 16:03:41.125192 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:41.125155    2561 patch_prober.go:28] interesting pod/image-registry-5c76476c85-rff5w container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:03:41.125373 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:41.125223    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" podUID="3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:03:42.535606 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:42.535572    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:42.535606 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:42.535615    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:03:43.709623 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:43.709595    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4xt6"
May 26 16:03:43.712791 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:43.712773    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-w25cq\""
May 26 16:03:43.720971 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:43.720954    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p4xt6"
May 26 16:03:43.846582 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:43.846544    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p4xt6"]
May 26 16:03:43.851339 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:03:43.851309    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c7e49d_53fc_4af3_874c_8a7b6ab588e7.slice/crio-386aef787cbee8458f2fe395e3d654918db92b85ed4239385cb57756b2995a41 WatchSource:0}: Error finding container 386aef787cbee8458f2fe395e3d654918db92b85ed4239385cb57756b2995a41: Status 404 returned error can't find the container with id 386aef787cbee8458f2fe395e3d654918db92b85ed4239385cb57756b2995a41
May 26 16:03:44.266955 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:44.266910    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4xt6" event={"ID":"78c7e49d-53fc-4af3-874c-8a7b6ab588e7","Type":"ContainerStarted","Data":"386aef787cbee8458f2fe395e3d654918db92b85ed4239385cb57756b2995a41"}
May 26 16:03:46.274307 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:46.274266    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4xt6" event={"ID":"78c7e49d-53fc-4af3-874c-8a7b6ab588e7","Type":"ContainerStarted","Data":"2dc113138be0c8be016f5fad583a0a197ea357e460b2a00c2b6c9724b105cbec"}
May 26 16:03:46.274307 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:46.274300    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p4xt6" event={"ID":"78c7e49d-53fc-4af3-874c-8a7b6ab588e7","Type":"ContainerStarted","Data":"d52f25ece9003588d5a0d8b0717287ebaf9af8bd506a312a3be283d2254743fe"}
May 26 16:03:46.274705 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:46.274407    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p4xt6"
May 26 16:03:46.297489 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:46.297442    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-p4xt6" podStartSLOduration=137.941945666 podStartE2EDuration="2m19.297428642s" podCreationTimestamp="2026-05-26 16:01:27 +0000 UTC" firstStartedPulling="2026-05-26 16:03:43.853304482 +0000 UTC m=+170.725562895" lastFinishedPulling="2026-05-26 16:03:45.208787441 +0000 UTC m=+172.081045871" observedRunningTime="2026-05-26 16:03:46.295857995 +0000 UTC m=+173.168116429" watchObservedRunningTime="2026-05-26 16:03:46.297428642 +0000 UTC m=+173.169687075"
May 26 16:03:49.078513 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.078481    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:49.078921 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.078558    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:49.080183 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.080160    2561 patch_prober.go:28] interesting pod/console-7dd78598cd-xk7tp container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused" start-of-body=
May 26 16:03:49.080304 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.080202    2561 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7dd78598cd-xk7tp" podUID="477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" containerName="console" probeResult="failure" output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused"
May 26 16:03:49.666903 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.666857    2561 patch_prober.go:28] interesting pod/image-registry-5c76476c85-rff5w container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:03:49.667083 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.666932    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" podUID="3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:03:49.667083 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.666976    2561 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:03:49.667472 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.667432    2561 kuberuntime_manager.go:1107] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"c8c39f22283e1f6bac23ff5ae914e40f878d6302eb9454ad39b8bf09a1f6f0bb"} pod="openshift-image-registry/image-registry-5c76476c85-rff5w" containerMessage="Container registry failed liveness probe, will be restarted"
May 26 16:03:49.671042 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.671011    2561 patch_prober.go:28] interesting pod/image-registry-5c76476c85-rff5w container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
May 26 16:03:49.671165 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:49.671079    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" podUID="3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 26 16:03:50.510914 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.510852    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-85c7d67fc8-dpwmn" podUID="f84bc018-3054-4e53-aee0-e146219cb9bb" containerName="console" containerID="cri-o://0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2" gracePeriod=15
May 26 16:03:50.752901 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.752861    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85c7d67fc8-dpwmn_f84bc018-3054-4e53-aee0-e146219cb9bb/console/0.log"
May 26 16:03:50.753012 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.752963    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:50.847754 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.847722    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-service-ca\") pod \"f84bc018-3054-4e53-aee0-e146219cb9bb\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") "
May 26 16:03:50.847938 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.847785    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-oauth-config\") pod \"f84bc018-3054-4e53-aee0-e146219cb9bb\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") "
May 26 16:03:50.847938 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.847826    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4w62\" (UniqueName: \"kubernetes.io/projected/f84bc018-3054-4e53-aee0-e146219cb9bb-kube-api-access-r4w62\") pod \"f84bc018-3054-4e53-aee0-e146219cb9bb\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") "
May 26 16:03:50.847938 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.847862    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-serving-cert\") pod \"f84bc018-3054-4e53-aee0-e146219cb9bb\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") "
May 26 16:03:50.847938 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.847918    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-console-config\") pod \"f84bc018-3054-4e53-aee0-e146219cb9bb\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") "
May 26 16:03:50.848179 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.847977    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-oauth-serving-cert\") pod \"f84bc018-3054-4e53-aee0-e146219cb9bb\" (UID: \"f84bc018-3054-4e53-aee0-e146219cb9bb\") "
May 26 16:03:50.848231 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.848180    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-service-ca" (OuterVolumeSpecName: "service-ca") pod "f84bc018-3054-4e53-aee0-e146219cb9bb" (UID: "f84bc018-3054-4e53-aee0-e146219cb9bb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:03:50.848514 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.848315    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:03:50.848514 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.848393    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-console-config" (OuterVolumeSpecName: "console-config") pod "f84bc018-3054-4e53-aee0-e146219cb9bb" (UID: "f84bc018-3054-4e53-aee0-e146219cb9bb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:03:50.848514 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.848428    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f84bc018-3054-4e53-aee0-e146219cb9bb" (UID: "f84bc018-3054-4e53-aee0-e146219cb9bb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:03:50.850305 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.850281    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84bc018-3054-4e53-aee0-e146219cb9bb-kube-api-access-r4w62" (OuterVolumeSpecName: "kube-api-access-r4w62") pod "f84bc018-3054-4e53-aee0-e146219cb9bb" (UID: "f84bc018-3054-4e53-aee0-e146219cb9bb"). InnerVolumeSpecName "kube-api-access-r4w62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:03:50.850305 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.850292    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f84bc018-3054-4e53-aee0-e146219cb9bb" (UID: "f84bc018-3054-4e53-aee0-e146219cb9bb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:03:50.850433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.850314    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f84bc018-3054-4e53-aee0-e146219cb9bb" (UID: "f84bc018-3054-4e53-aee0-e146219cb9bb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:03:50.949619 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.949587    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-oauth-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:03:50.949619 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.949615    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4w62\" (UniqueName: \"kubernetes.io/projected/f84bc018-3054-4e53-aee0-e146219cb9bb-kube-api-access-r4w62\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:03:50.949619 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.949625    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84bc018-3054-4e53-aee0-e146219cb9bb-console-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:03:50.949842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.949634    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-console-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:03:50.949842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:50.949643    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84bc018-3054-4e53-aee0-e146219cb9bb-oauth-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:03:51.292971 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.292934    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85c7d67fc8-dpwmn_f84bc018-3054-4e53-aee0-e146219cb9bb/console/0.log"
May 26 16:03:51.293133 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.292983    2561 generic.go:358] "Generic (PLEG): container finished" podID="f84bc018-3054-4e53-aee0-e146219cb9bb" containerID="0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2" exitCode=2
May 26 16:03:51.293133 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.293065    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c7d67fc8-dpwmn" event={"ID":"f84bc018-3054-4e53-aee0-e146219cb9bb","Type":"ContainerDied","Data":"0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2"}
May 26 16:03:51.293133 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.293096    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c7d67fc8-dpwmn" event={"ID":"f84bc018-3054-4e53-aee0-e146219cb9bb","Type":"ContainerDied","Data":"887e1afe9d48366b57839c082327939c97e271ea5af658e162c2776e51cf8b37"}
May 26 16:03:51.293133 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.293124    2561 scope.go:117] "RemoveContainer" containerID="0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2"
May 26 16:03:51.293342 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.293334    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c7d67fc8-dpwmn"
May 26 16:03:51.304209 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.304167    2561 scope.go:117] "RemoveContainer" containerID="0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2"
May 26 16:03:51.304663 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:03:51.304610    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2\": container with ID starting with 0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2 not found: ID does not exist" containerID="0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2"
May 26 16:03:51.304757 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.304672    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2"} err="failed to get container status \"0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2\": rpc error: code = NotFound desc = could not find container \"0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2\": container with ID starting with 0aba0443fc4d4cbd04b645cee4cb84f4a13a9de491475e030cc614d369dbbfd2 not found: ID does not exist"
May 26 16:03:51.317710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.317687    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c7d67fc8-dpwmn"]
May 26 16:03:51.321190 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.321166    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c7d67fc8-dpwmn"]
May 26 16:03:51.711363 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:51.711284    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f84bc018-3054-4e53-aee0-e146219cb9bb" path="/var/lib/kubelet/pods/f84bc018-3054-4e53-aee0-e146219cb9bb/volumes"
May 26 16:03:56.280866 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:56.280837    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p4xt6"
May 26 16:03:59.082098 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:59.082064    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:59.086038 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:59.086012    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:03:59.672324 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:03:59.672297    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:04:02.541000 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:02.540972    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:04:02.544825 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:02.544802    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-59874dc46b-hvndz"
May 26 16:04:11.353598 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:11.353510    2561 generic.go:358] "Generic (PLEG): container finished" podID="3d007b98-2dc2-43a6-9083-5cdbda352f5e" containerID="21dfaefe5dfd7ac36e2e179f9c2b2519ac8e90eaa99b3212de9232f7163c3ecc" exitCode=0
May 26 16:04:11.353598 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:11.353585    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv" event={"ID":"3d007b98-2dc2-43a6-9083-5cdbda352f5e","Type":"ContainerDied","Data":"21dfaefe5dfd7ac36e2e179f9c2b2519ac8e90eaa99b3212de9232f7163c3ecc"}
May 26 16:04:11.354036 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:11.353921    2561 scope.go:117] "RemoveContainer" containerID="21dfaefe5dfd7ac36e2e179f9c2b2519ac8e90eaa99b3212de9232f7163c3ecc"
May 26 16:04:12.358578 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:12.358539    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-qwmwv" event={"ID":"3d007b98-2dc2-43a6-9083-5cdbda352f5e","Type":"ContainerStarted","Data":"7b9c2d8f01c4b30523d0cf23ccf557d5276fdecda384a2ee7c971d1660d23f9f"}
May 26 16:04:14.686155 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:14.686109    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" podUID="3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c" containerName="registry" containerID="cri-o://c8c39f22283e1f6bac23ff5ae914e40f878d6302eb9454ad39b8bf09a1f6f0bb" gracePeriod=30
May 26 16:04:16.372294 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:16.372257    2561 generic.go:358] "Generic (PLEG): container finished" podID="3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c" containerID="c8c39f22283e1f6bac23ff5ae914e40f878d6302eb9454ad39b8bf09a1f6f0bb" exitCode=0
May 26 16:04:16.372704 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:16.372317    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" event={"ID":"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c","Type":"ContainerDied","Data":"c8c39f22283e1f6bac23ff5ae914e40f878d6302eb9454ad39b8bf09a1f6f0bb"}
May 26 16:04:16.372704 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:16.372339    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5c76476c85-rff5w" event={"ID":"3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c","Type":"ContainerStarted","Data":"03219c08caa6f431f959fb3166767785c00c1b9d1773c5acadcb051d635e2c75"}
May 26 16:04:16.372704 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:16.372419    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:04:18.400439 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:18.400396    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c76476c85-rff5w_3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c/registry/0.log"
May 26 16:04:18.406071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:18.406041    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c76476c85-rff5w_3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c/registry/1.log"
May 26 16:04:18.429157 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:18.429137    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wj5zg_3f685ebf-142e-4075-888a-d44a456c5812/node-ca/0.log"
May 26 16:04:24.388069 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:24.388021    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:24.404110 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:24.404077    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:24.419658 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:24.419634    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:37.379769 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:37.379737    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5c76476c85-rff5w"
May 26 16:04:38.195664 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.195625    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:04:38.196123 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.196077    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="alertmanager" containerID="cri-o://7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7" gracePeriod=120
May 26 16:04:38.196200 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.196151    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy-metric" containerID="cri-o://fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80" gracePeriod=120
May 26 16:04:38.196253 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.196184    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="prom-label-proxy" containerID="cri-o://0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985" gracePeriod=120
May 26 16:04:38.196253 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.196201    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy-web" containerID="cri-o://b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2" gracePeriod=120
May 26 16:04:38.196360 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.196236    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy" containerID="cri-o://4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765" gracePeriod=120
May 26 16:04:38.196360 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.196237    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="config-reloader" containerID="cri-o://be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6" gracePeriod=120
May 26 16:04:38.446196 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.446106    2561 generic.go:358] "Generic (PLEG): container finished" podID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerID="0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985" exitCode=0
May 26 16:04:38.446196 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.446130    2561 generic.go:358] "Generic (PLEG): container finished" podID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerID="4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765" exitCode=0
May 26 16:04:38.446196 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.446136    2561 generic.go:358] "Generic (PLEG): container finished" podID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerID="be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6" exitCode=0
May 26 16:04:38.446196 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.446142    2561 generic.go:358] "Generic (PLEG): container finished" podID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerID="7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7" exitCode=0
May 26 16:04:38.446612 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.446184    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerDied","Data":"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985"}
May 26 16:04:38.446612 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.446222    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerDied","Data":"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765"}
May 26 16:04:38.446612 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.446233    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerDied","Data":"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6"}
May 26 16:04:38.446612 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:38.446245    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerDied","Data":"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7"}
May 26 16:04:39.443510 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.443479    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:39.451736 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.451710    2561 generic.go:358] "Generic (PLEG): container finished" podID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerID="fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80" exitCode=0
May 26 16:04:39.451736 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.451733    2561 generic.go:358] "Generic (PLEG): container finished" podID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerID="b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2" exitCode=0
May 26 16:04:39.452130 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.451792    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerDied","Data":"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80"}
May 26 16:04:39.452130 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.451814    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 26 16:04:39.452130 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.451833    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerDied","Data":"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2"}
May 26 16:04:39.452130 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.451851    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3325d4c-ac99-44b4-b576-eae4b16b3830","Type":"ContainerDied","Data":"b66db5d891798dc37b83e5263b7e469f4255e3e32185f6b4baf032f0135de6f4"}
May 26 16:04:39.452130 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.451870    2561 scope.go:117] "RemoveContainer" containerID="0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985"
May 26 16:04:39.458913 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.458868    2561 scope.go:117] "RemoveContainer" containerID="fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80"
May 26 16:04:39.465434 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.465415    2561 scope.go:117] "RemoveContainer" containerID="4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765"
May 26 16:04:39.473842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.473292    2561 scope.go:117] "RemoveContainer" containerID="b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2"
May 26 16:04:39.483002 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.482980    2561 scope.go:117] "RemoveContainer" containerID="be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6"
May 26 16:04:39.490930 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.490870    2561 scope.go:117] "RemoveContainer" containerID="7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7"
May 26 16:04:39.498292 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.498275    2561 scope.go:117] "RemoveContainer" containerID="0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188"
May 26 16:04:39.506760 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.506735    2561 scope.go:117] "RemoveContainer" containerID="0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985"
May 26 16:04:39.507059 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:39.507039    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985\": container with ID starting with 0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985 not found: ID does not exist" containerID="0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985"
May 26 16:04:39.507133 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.507070    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985"} err="failed to get container status \"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985\": rpc error: code = NotFound desc = could not find container \"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985\": container with ID starting with 0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985 not found: ID does not exist"
May 26 16:04:39.507133 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.507092    2561 scope.go:117] "RemoveContainer" containerID="fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80"
May 26 16:04:39.507328 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:39.507314    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80\": container with ID starting with fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80 not found: ID does not exist" containerID="fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80"
May 26 16:04:39.507373 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.507331    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80"} err="failed to get container status \"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80\": rpc error: code = NotFound desc = could not find container \"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80\": container with ID starting with fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80 not found: ID does not exist"
May 26 16:04:39.507373 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.507343    2561 scope.go:117] "RemoveContainer" containerID="4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765"
May 26 16:04:39.507567 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:39.507543    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765\": container with ID starting with 4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765 not found: ID does not exist" containerID="4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765"
May 26 16:04:39.507623 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.507566    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765"} err="failed to get container status \"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765\": rpc error: code = NotFound desc = could not find container \"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765\": container with ID starting with 4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765 not found: ID does not exist"
May 26 16:04:39.507623 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.507584    2561 scope.go:117] "RemoveContainer" containerID="b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2"
May 26 16:04:39.507787 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:39.507767    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2\": container with ID starting with b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2 not found: ID does not exist" containerID="b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2"
May 26 16:04:39.507841 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.507792    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2"} err="failed to get container status \"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2\": rpc error: code = NotFound desc = could not find container \"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2\": container with ID starting with b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2 not found: ID does not exist"
May 26 16:04:39.507841 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.507806    2561 scope.go:117] "RemoveContainer" containerID="be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6"
May 26 16:04:39.508121 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:39.508103    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6\": container with ID starting with be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6 not found: ID does not exist" containerID="be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6"
May 26 16:04:39.508192 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.508126    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6"} err="failed to get container status \"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6\": rpc error: code = NotFound desc = could not find container \"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6\": container with ID starting with be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6 not found: ID does not exist"
May 26 16:04:39.508192 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.508140    2561 scope.go:117] "RemoveContainer" containerID="7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7"
May 26 16:04:39.508378 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:39.508358    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7\": container with ID starting with 7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7 not found: ID does not exist" containerID="7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7"
May 26 16:04:39.508493 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.508387    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7"} err="failed to get container status \"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7\": rpc error: code = NotFound desc = could not find container \"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7\": container with ID starting with 7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7 not found: ID does not exist"
May 26 16:04:39.508493 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.508415    2561 scope.go:117] "RemoveContainer" containerID="0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188"
May 26 16:04:39.508677 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:39.508660    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188\": container with ID starting with 0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188 not found: ID does not exist" containerID="0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188"
May 26 16:04:39.508717 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.508682    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188"} err="failed to get container status \"0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188\": rpc error: code = NotFound desc = could not find container \"0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188\": container with ID starting with 0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188 not found: ID does not exist"
May 26 16:04:39.508717 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.508698    2561 scope.go:117] "RemoveContainer" containerID="0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985"
May 26 16:04:39.508940 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.508924    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985"} err="failed to get container status \"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985\": rpc error: code = NotFound desc = could not find container \"0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985\": container with ID starting with 0f305f3774610b50b5de6618fe02c5d6db04861853dcff0a48fcfe61f09a3985 not found: ID does not exist"
May 26 16:04:39.508940 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.508940    2561 scope.go:117] "RemoveContainer" containerID="fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80"
May 26 16:04:39.509170 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.509145    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80"} err="failed to get container status \"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80\": rpc error: code = NotFound desc = could not find container \"fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80\": container with ID starting with fb067d05e68ccbe002034e5b4e135150672169453abe8898b8444b1133405e80 not found: ID does not exist"
May 26 16:04:39.509243 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.509173    2561 scope.go:117] "RemoveContainer" containerID="4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765"
May 26 16:04:39.509369 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.509352    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765"} err="failed to get container status \"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765\": rpc error: code = NotFound desc = could not find container \"4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765\": container with ID starting with 4257380826a6771c94675cbfab551c9b5a578f6eb2b54ba93d9cb66189753765 not found: ID does not exist"
May 26 16:04:39.509434 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.509371    2561 scope.go:117] "RemoveContainer" containerID="b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2"
May 26 16:04:39.509582 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.509562    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2"} err="failed to get container status \"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2\": rpc error: code = NotFound desc = could not find container \"b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2\": container with ID starting with b3c82bdd556d3ae00ecddf1fb03d6e36db8d4a8bf946d7b96e70460b9088d1b2 not found: ID does not exist"
May 26 16:04:39.509651 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.509582    2561 scope.go:117] "RemoveContainer" containerID="be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6"
May 26 16:04:39.509782 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.509763    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6"} err="failed to get container status \"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6\": rpc error: code = NotFound desc = could not find container \"be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6\": container with ID starting with be9f0136e033a13413d7e6fd350ef6267b82c5740dd18506a948753d3d1f2ea6 not found: ID does not exist"
May 26 16:04:39.509857 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.509783    2561 scope.go:117] "RemoveContainer" containerID="7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7"
May 26 16:04:39.510114 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.510095    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7"} err="failed to get container status \"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7\": rpc error: code = NotFound desc = could not find container \"7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7\": container with ID starting with 7b74615d49f46883d92b2d81140f862d45bb1d02a60bcab40b5ec369db13f2c7 not found: ID does not exist"
May 26 16:04:39.510182 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.510114    2561 scope.go:117] "RemoveContainer" containerID="0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188"
May 26 16:04:39.510348 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.510333    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188"} err="failed to get container status \"0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188\": rpc error: code = NotFound desc = could not find container \"0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188\": container with ID starting with 0f54a5baac25f33c66721c0f1ecbfab26ff3f9ed770f9cd707e77b206ade8188 not found: ID does not exist"
May 26 16:04:39.584153 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584114    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584314 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584166    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-web-config\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584314 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584197    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-trusted-ca-bundle\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584314 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584232    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584314 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584281    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-volume\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584314 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584308    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-metrics-client-ca\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584564 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584350    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-main-tls\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584564 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584384    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-web\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584700 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584664    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:39.584829 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584802    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqlpm\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-kube-api-access-tqlpm\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584923 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584912    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-tls-assets\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584939    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-out\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.584993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584971    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-main-db\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.585099 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.584999    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-cluster-tls-config\") pod \"c3325d4c-ac99-44b4-b576-eae4b16b3830\" (UID: \"c3325d4c-ac99-44b4-b576-eae4b16b3830\") "
May 26 16:04:39.585099 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.585036    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:39.585574 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.585309    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.585574 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.585334    2561 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3325d4c-ac99-44b4-b576-eae4b16b3830-metrics-client-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.585798 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.585746    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:04:39.587038 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.587010    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-volume" (OuterVolumeSpecName: "config-volume") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:39.587483 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.587456    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:39.587725 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.587695    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:39.587806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.587720    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:39.588004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.587971    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:39.588494 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.588474    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:39.588803 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.588781    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-kube-api-access-tqlpm" (OuterVolumeSpecName: "kube-api-access-tqlpm") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "kube-api-access-tqlpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:39.589090 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.589066    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-out" (OuterVolumeSpecName: "config-out") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:04:39.592567 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.592541    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:39.598445 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.598422    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-web-config" (OuterVolumeSpecName: "web-config") pod "c3325d4c-ac99-44b4-b576-eae4b16b3830" (UID: "c3325d4c-ac99-44b4-b576-eae4b16b3830"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:39.686152 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686120    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-tls-assets\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686152 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686151    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-out\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686165    2561 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3325d4c-ac99-44b4-b576-eae4b16b3830-alertmanager-main-db\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686178    2561 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-cluster-tls-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686192    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686205    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-web-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686218    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686230    2561 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-config-volume\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686244    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-main-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686257    2561 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3325d4c-ac99-44b4-b576-eae4b16b3830-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.686364 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.686270    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqlpm\" (UniqueName: \"kubernetes.io/projected/c3325d4c-ac99-44b4-b576-eae4b16b3830-kube-api-access-tqlpm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:39.771168 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.771138    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:04:39.775085 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:39.775055    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 26 16:04:41.711857 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:41.711818    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" path="/var/lib/kubelet/pods/c3325d4c-ac99-44b4-b576-eae4b16b3830/volumes"
May 26 16:04:42.424164 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:42.424131    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:42.424584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:42.424543    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="prometheus" containerID="cri-o://0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf" gracePeriod=600
May 26 16:04:42.424584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:42.424565    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy" containerID="cri-o://be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42" gracePeriod=600
May 26 16:04:42.424806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:42.424585    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="thanos-sidecar" containerID="cri-o://41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97" gracePeriod=600
May 26 16:04:42.424806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:42.424609    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy-web" containerID="cri-o://8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47" gracePeriod=600
May 26 16:04:42.424806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:42.424626    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy-thanos" containerID="cri-o://784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6" gracePeriod=600
May 26 16:04:42.424806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:42.424697    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="config-reloader" containerID="cri-o://c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546" gracePeriod=600
May 26 16:04:43.468787 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468752    2561 generic.go:358] "Generic (PLEG): container finished" podID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerID="784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6" exitCode=0
May 26 16:04:43.468787 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468777    2561 generic.go:358] "Generic (PLEG): container finished" podID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerID="be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42" exitCode=0
May 26 16:04:43.468787 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468784    2561 generic.go:358] "Generic (PLEG): container finished" podID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerID="41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97" exitCode=0
May 26 16:04:43.468787 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468789    2561 generic.go:358] "Generic (PLEG): container finished" podID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerID="c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546" exitCode=0
May 26 16:04:43.468787 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468794    2561 generic.go:358] "Generic (PLEG): container finished" podID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerID="0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf" exitCode=0
May 26 16:04:43.469331 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468826    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerDied","Data":"784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6"}
May 26 16:04:43.469331 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468867    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerDied","Data":"be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42"}
May 26 16:04:43.469331 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468882    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerDied","Data":"41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97"}
May 26 16:04:43.469331 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468915    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerDied","Data":"c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546"}
May 26 16:04:43.469331 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.468929    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerDied","Data":"0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf"}
May 26 16:04:43.671799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.671775    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:43.718613 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718578    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-kube-rbac-proxy\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.718776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718628    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-serving-certs-ca-bundle\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.718776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718661    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-trusted-ca-bundle\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.718776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718705    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-rulefiles-0\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.718776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718731    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-config-out\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718793    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-tls-assets\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718829    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-tls\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718860    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-web-config\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718910    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718956    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-config\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.718980    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-thanos-prometheus-http-client-file\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719013    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-kubelet-serving-ca-bundle\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719038    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-grpc-tls\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719063    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-metrics-client-certs\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719094    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719121    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:43.719433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719129    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwnbx\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-kube-api-access-fwnbx\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719148    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:43.719433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719190    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-metrics-client-ca\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719433 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719226    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-db\") pod \"8c2732de-3d29-4700-bf6c-71f1522ef58d\" (UID: \"8c2732de-3d29-4700-bf6c-71f1522ef58d\") "
May 26 16:04:43.719951 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719510    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.719951 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.719531    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-trusted-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.723073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.720654    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:04:43.723073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.721189    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:43.723073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.721268    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:43.723073 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.721514    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.723836 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.723656    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.723981 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.723836    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-config-out" (OuterVolumeSpecName: "config-out") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:04:43.724495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.724048    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.724495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.724301    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.724495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.724430    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.724495 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.724487    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:04:43.724860 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.724823    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.724971 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.724918    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.725038 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.725007    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-kube-api-access-fwnbx" (OuterVolumeSpecName: "kube-api-access-fwnbx") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "kube-api-access-fwnbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:43.726140 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.726120    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-config" (OuterVolumeSpecName: "config") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.726436 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.726403    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:04:43.736638 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.736617    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-web-config" (OuterVolumeSpecName: "web-config") pod "8c2732de-3d29-4700-bf6c-71f1522ef58d" (UID: "8c2732de-3d29-4700-bf6c-71f1522ef58d"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:04:43.820367 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820331    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-kube-rbac-proxy\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820367 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820363    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820367 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820374    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-config-out\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820384    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-tls-assets\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820392    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820400    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-web-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820411    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820420    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820429    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-thanos-prometheus-http-client-file\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820439    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820448    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-grpc-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820457    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-metrics-client-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820466    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8c2732de-3d29-4700-bf6c-71f1522ef58d-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820474    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwnbx\" (UniqueName: \"kubernetes.io/projected/8c2732de-3d29-4700-bf6c-71f1522ef58d-kube-api-access-fwnbx\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820484    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c2732de-3d29-4700-bf6c-71f1522ef58d-configmap-metrics-client-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:43.820584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:43.820494    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8c2732de-3d29-4700-bf6c-71f1522ef58d-prometheus-k8s-db\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:04:44.474445 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.474413    2561 generic.go:358] "Generic (PLEG): container finished" podID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerID="8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47" exitCode=0
May 26 16:04:44.474832 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.474493    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerDied","Data":"8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47"}
May 26 16:04:44.474832 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.474534    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.474832 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.474548    2561 scope.go:117] "RemoveContainer" containerID="784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6"
May 26 16:04:44.474832 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.474536    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8c2732de-3d29-4700-bf6c-71f1522ef58d","Type":"ContainerDied","Data":"72f08609a52554590447fe7af270ba9ba53bef09d70bfa34aac7fed700f6c67b"}
May 26 16:04:44.482540 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.482523    2561 scope.go:117] "RemoveContainer" containerID="be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42"
May 26 16:04:44.489684 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.489667    2561 scope.go:117] "RemoveContainer" containerID="8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47"
May 26 16:04:44.495973 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.495954    2561 scope.go:117] "RemoveContainer" containerID="41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97"
May 26 16:04:44.500134 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.500105    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:44.503576 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.503549    2561 scope.go:117] "RemoveContainer" containerID="c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546"
May 26 16:04:44.506171 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.506151    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:44.514816 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.514799    2561 scope.go:117] "RemoveContainer" containerID="0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf"
May 26 16:04:44.521730 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.521710    2561 scope.go:117] "RemoveContainer" containerID="037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d"
May 26 16:04:44.528020 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.527998    2561 scope.go:117] "RemoveContainer" containerID="784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6"
May 26 16:04:44.528289 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:44.528269    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6\": container with ID starting with 784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6 not found: ID does not exist" containerID="784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6"
May 26 16:04:44.528333 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.528299    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6"} err="failed to get container status \"784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6\": rpc error: code = NotFound desc = could not find container \"784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6\": container with ID starting with 784d8cc624f57f75bac0bbb6b67374fca378f0e5ff389bde5e4b1e93acd65be6 not found: ID does not exist"
May 26 16:04:44.528333 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.528320    2561 scope.go:117] "RemoveContainer" containerID="be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42"
May 26 16:04:44.528547 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:44.528529    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42\": container with ID starting with be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42 not found: ID does not exist" containerID="be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42"
May 26 16:04:44.528611 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.528560    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42"} err="failed to get container status \"be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42\": rpc error: code = NotFound desc = could not find container \"be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42\": container with ID starting with be12be39bc9e398bf9ed37bd0b139fb775f511185f787232d5e035d064d6ae42 not found: ID does not exist"
May 26 16:04:44.528611 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.528586    2561 scope.go:117] "RemoveContainer" containerID="8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47"
May 26 16:04:44.528845 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:44.528829    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47\": container with ID starting with 8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47 not found: ID does not exist" containerID="8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47"
May 26 16:04:44.528910 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.528858    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47"} err="failed to get container status \"8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47\": rpc error: code = NotFound desc = could not find container \"8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47\": container with ID starting with 8fb179b4d11fd0dfe0cd1f9de2301651ac88699f0b019f7a74f17ca11f5b4f47 not found: ID does not exist"
May 26 16:04:44.528910 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.528874    2561 scope.go:117] "RemoveContainer" containerID="41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97"
May 26 16:04:44.529088 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:44.529070    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97\": container with ID starting with 41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97 not found: ID does not exist" containerID="41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97"
May 26 16:04:44.529125 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.529094    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97"} err="failed to get container status \"41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97\": rpc error: code = NotFound desc = could not find container \"41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97\": container with ID starting with 41a69a1408f3a0769a1abb145ae72ea48e132750d153f6b8cf860372d0eaec97 not found: ID does not exist"
May 26 16:04:44.529125 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.529107    2561 scope.go:117] "RemoveContainer" containerID="c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546"
May 26 16:04:44.529320 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:44.529306    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546\": container with ID starting with c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546 not found: ID does not exist" containerID="c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546"
May 26 16:04:44.529357 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.529325    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546"} err="failed to get container status \"c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546\": rpc error: code = NotFound desc = could not find container \"c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546\": container with ID starting with c350e0d6187a2824d3f88e4a395ff416051adbe8a67b177592add8b6b7c9c546 not found: ID does not exist"
May 26 16:04:44.529357 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.529337    2561 scope.go:117] "RemoveContainer" containerID="0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf"
May 26 16:04:44.529514 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:44.529492    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf\": container with ID starting with 0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf not found: ID does not exist" containerID="0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf"
May 26 16:04:44.529514 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.529520    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf"} err="failed to get container status \"0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf\": rpc error: code = NotFound desc = could not find container \"0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf\": container with ID starting with 0f55ade24901165df27bd439b2a43d81f94e5b2a603b4ee810713e452a41bbbf not found: ID does not exist"
May 26 16:04:44.529626 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.529541    2561 scope.go:117] "RemoveContainer" containerID="037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d"
May 26 16:04:44.529810 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:04:44.529786    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d\": container with ID starting with 037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d not found: ID does not exist" containerID="037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d"
May 26 16:04:44.529958 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.529818    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d"} err="failed to get container status \"037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d\": rpc error: code = NotFound desc = could not find container \"037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d\": container with ID starting with 037dd6b2c22bfd794e5f7fc406665cc1dfd193ae88d7885d728b66d549e3c95d not found: ID does not exist"
May 26 16:04:44.531580 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531561    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:44.531842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531824    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="prom-label-proxy"
May 26 16:04:44.531842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531842    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="prom-label-proxy"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531853    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy-thanos"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531858    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy-thanos"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531867    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="thanos-sidecar"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531872    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="thanos-sidecar"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531877    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="alertmanager"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531882    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="alertmanager"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531901    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy-web"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531909    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy-web"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531919    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531923    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531932    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="prometheus"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531937    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="prometheus"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531943    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84bc018-3054-4e53-aee0-e146219cb9bb" containerName="console"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531948    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84bc018-3054-4e53-aee0-e146219cb9bb" containerName="console"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531953    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="config-reloader"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531957    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="config-reloader"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531964    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy-web"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531969    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy-web"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531980    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy-metric"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531988    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy-metric"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.531995    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="config-reloader"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532000    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="config-reloader"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532006    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="init-config-reloader"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532012    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="init-config-reloader"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532017    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy"
May 26 16:04:44.532004 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532023    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532028    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="init-config-reloader"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532033    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="init-config-reloader"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532078    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="thanos-sidecar"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532088    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532094    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="config-reloader"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532101    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy-web"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532106    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="f84bc018-3054-4e53-aee0-e146219cb9bb" containerName="console"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532112    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy-metric"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532118    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532123    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="kube-rbac-proxy-web"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532130    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="config-reloader"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532139    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="prom-label-proxy"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532145    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="prometheus"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532151    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="c3325d4c-ac99-44b4-b576-eae4b16b3830" containerName="alertmanager"
May 26 16:04:44.532799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.532156    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" containerName="kube-rbac-proxy-thanos"
May 26 16:04:44.535837 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.535822    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.538805 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.538783    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 26 16:04:44.538932 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.538805    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 26 16:04:44.538932 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.538784    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 26 16:04:44.538932 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.538784    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 26 16:04:44.539375 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.538830    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-knhsq\""
May 26 16:04:44.540163 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.540144    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 26 16:04:44.540330 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.540309    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 26 16:04:44.540419 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.540402    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 26 16:04:44.541857 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.541840    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 26 16:04:44.542206 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.542031    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5bl6nro5d0n34\""
May 26 16:04:44.542506 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.542492    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 26 16:04:44.543087 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.543073    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 26 16:04:44.545380 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.545308    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 26 16:04:44.548862 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.548841    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 26 16:04:44.549924 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.549903    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:44.626582 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626553    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626582 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626585    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626656    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626696    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b9b463f-5708-4999-990f-ad5d7be3c697-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626743    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626756    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626979 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626794    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626979 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626830    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-config\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626979 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626846    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626979 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626873    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626979 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626911    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626979 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.626979 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.626971    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.627171 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.627010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b9b463f-5708-4999-990f-ad5d7be3c697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.627171 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.627028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66m8b\" (UniqueName: \"kubernetes.io/projected/3b9b463f-5708-4999-990f-ad5d7be3c697-kube-api-access-66m8b\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.627171 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.627052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.727656 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.727579    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-config\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.727656 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.727616    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.727782 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.727764    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.727819 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.727802    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.727864 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.727843    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.727923 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.727870    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.727980 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.727929    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b9b463f-5708-4999-990f-ad5d7be3c697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.727980 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.727958    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-66m8b\" (UniqueName: \"kubernetes.io/projected/3b9b463f-5708-4999-990f-ad5d7be3c697-kube-api-access-66m8b\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728143 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728121    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728274 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728255    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728293    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728349 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728320    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728452 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728350    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728452 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728416    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728567 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728444    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b9b463f-5708-4999-990f-ad5d7be3c697-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728567 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728497    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728567 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728529    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728768 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728566    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728768 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728631    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.728768 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.728743    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.730778 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.730752    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.730873 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.730757    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.731259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.731044    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.731259 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.731197    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.731411 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.731362    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-web-config\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.731468 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.731405    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.731468 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.731420    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3b9b463f-5708-4999-990f-ad5d7be3c697-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.731824 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.731574    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.731824 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.731786    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3b9b463f-5708-4999-990f-ad5d7be3c697-config-out\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.731999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.731947    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.732488 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.732448    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-config\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.733649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.733629    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.733731 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.733668    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.733900 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.733863    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3b9b463f-5708-4999-990f-ad5d7be3c697-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.734679 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.734662    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3b9b463f-5708-4999-990f-ad5d7be3c697-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.740489 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.740464    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66m8b\" (UniqueName: \"kubernetes.io/projected/3b9b463f-5708-4999-990f-ad5d7be3c697-kube-api-access-66m8b\") pod \"prometheus-k8s-0\" (UID: \"3b9b463f-5708-4999-990f-ad5d7be3c697\") " pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:44.851905 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:44.851838    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:04:45.002170 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:45.002141    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 26 16:04:45.004534 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:04:45.004498    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9b463f_5708_4999_990f_ad5d7be3c697.slice/crio-5a0ba09fd2b5f6ac516854a6f23eda06f8b28f9a0244b489493963acd7a9fe52 WatchSource:0}: Error finding container 5a0ba09fd2b5f6ac516854a6f23eda06f8b28f9a0244b489493963acd7a9fe52: Status 404 returned error can't find the container with id 5a0ba09fd2b5f6ac516854a6f23eda06f8b28f9a0244b489493963acd7a9fe52
May 26 16:04:45.478871 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:45.478836    2561 generic.go:358] "Generic (PLEG): container finished" podID="3b9b463f-5708-4999-990f-ad5d7be3c697" containerID="79fdd1bad61ce7b9af94a9e0c38a6484f5986437c659b864e1b817f45536a270" exitCode=0
May 26 16:04:45.479245 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:45.478906    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b9b463f-5708-4999-990f-ad5d7be3c697","Type":"ContainerDied","Data":"79fdd1bad61ce7b9af94a9e0c38a6484f5986437c659b864e1b817f45536a270"}
May 26 16:04:45.479245 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:45.478931    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b9b463f-5708-4999-990f-ad5d7be3c697","Type":"ContainerStarted","Data":"5a0ba09fd2b5f6ac516854a6f23eda06f8b28f9a0244b489493963acd7a9fe52"}
May 26 16:04:45.713028 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:45.712992    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2732de-3d29-4700-bf6c-71f1522ef58d" path="/var/lib/kubelet/pods/8c2732de-3d29-4700-bf6c-71f1522ef58d/volumes"
May 26 16:04:46.484868 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:46.484835    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b9b463f-5708-4999-990f-ad5d7be3c697","Type":"ContainerStarted","Data":"451c00ef9984b2f720179b054dbe7a09443e78331d6ccf785760eb787ca6ad59"}
May 26 16:04:46.484868 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:46.484870    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b9b463f-5708-4999-990f-ad5d7be3c697","Type":"ContainerStarted","Data":"0aeb295dc7ecbf706559307887d221fe9c6b3a46468c1a78a9dac48f01eb29d1"}
May 26 16:04:46.485289 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:46.484881    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b9b463f-5708-4999-990f-ad5d7be3c697","Type":"ContainerStarted","Data":"16e6ee5a5058b0eddce0ea60d6676e937ea0853aadffe37b1ffc4218484c899a"}
May 26 16:04:46.485289 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:46.484908    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b9b463f-5708-4999-990f-ad5d7be3c697","Type":"ContainerStarted","Data":"421a03350819864adc959d58deda54a4c147599e43b2ffb58f31d2216a1ab0e2"}
May 26 16:04:46.485289 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:46.484916    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b9b463f-5708-4999-990f-ad5d7be3c697","Type":"ContainerStarted","Data":"b6252518ce9416546345c1da39b803c23aa04b12e0e4d194d3c907a231411175"}
May 26 16:04:46.485289 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:46.484924    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3b9b463f-5708-4999-990f-ad5d7be3c697","Type":"ContainerStarted","Data":"f90bff7067c3e6e8602e54b83ec49b3d2c44e972294d57ebdaa94257108305a9"}
May 26 16:04:46.521299 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:46.521256    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.521237983 podStartE2EDuration="2.521237983s" podCreationTimestamp="2026-05-26 16:04:44 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:04:46.51891054 +0000 UTC m=+233.391168978" watchObservedRunningTime="2026-05-26 16:04:46.521237983 +0000 UTC m=+233.393496416"
May 26 16:04:47.857332 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:47.857298    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dd78598cd-xk7tp"]
May 26 16:04:49.852325 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:04:49.852288    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:12.876190 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:12.876072    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7dd78598cd-xk7tp" podUID="477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" containerName="console" containerID="cri-o://ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f" gracePeriod=15
May 26 16:05:13.125408 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.125383    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7dd78598cd-xk7tp_477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53/console/0.log"
May 26 16:05:13.125573 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.125444    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:05:13.272494 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272470    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-trusted-ca-bundle\") pod \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") "
May 26 16:05:13.272649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272502    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-oauth-serving-cert\") pod \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") "
May 26 16:05:13.272649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272528    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-config\") pod \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") "
May 26 16:05:13.272649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272550    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-service-ca\") pod \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") "
May 26 16:05:13.272649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272572    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-serving-cert\") pod \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") "
May 26 16:05:13.272649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272624    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znvbm\" (UniqueName: \"kubernetes.io/projected/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-kube-api-access-znvbm\") pod \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") "
May 26 16:05:13.272882 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272659    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-oauth-config\") pod \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\" (UID: \"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53\") "
May 26 16:05:13.272989 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272964    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" (UID: "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:13.273050 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272988    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" (UID: "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:13.273050 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.272993    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-service-ca" (OuterVolumeSpecName: "service-ca") pod "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" (UID: "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:13.273292 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.273251    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-config" (OuterVolumeSpecName: "console-config") pod "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" (UID: "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 26 16:05:13.274873 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.274847    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" (UID: "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:13.274990 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.274867    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" (UID: "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:05:13.274990 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.274917    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-kube-api-access-znvbm" (OuterVolumeSpecName: "kube-api-access-znvbm") pod "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" (UID: "477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53"). InnerVolumeSpecName "kube-api-access-znvbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:05:13.374035 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.373988    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-trusted-ca-bundle\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:05:13.374035 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.374030    2561 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-oauth-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:05:13.374035 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.374042    2561 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:05:13.374035 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.374051    2561 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-service-ca\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:05:13.374287 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.374059    2561 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-serving-cert\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:05:13.374287 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.374068    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znvbm\" (UniqueName: \"kubernetes.io/projected/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-kube-api-access-znvbm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:05:13.374287 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.374077    2561 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53-console-oauth-config\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:05:13.564464 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.564391    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7dd78598cd-xk7tp_477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53/console/0.log"
May 26 16:05:13.564464 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.564429    2561 generic.go:358] "Generic (PLEG): container finished" podID="477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" containerID="ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f" exitCode=2
May 26 16:05:13.564635 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.564491    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd78598cd-xk7tp" event={"ID":"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53","Type":"ContainerDied","Data":"ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f"}
May 26 16:05:13.564635 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.564497    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dd78598cd-xk7tp"
May 26 16:05:13.564635 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.564523    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dd78598cd-xk7tp" event={"ID":"477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53","Type":"ContainerDied","Data":"cffd5a5e50a451f2538377edefe5a2cb7b8ebc38b0793ff8f08d211464f1ec79"}
May 26 16:05:13.564635 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.564538    2561 scope.go:117] "RemoveContainer" containerID="ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f"
May 26 16:05:13.574040 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.573951    2561 scope.go:117] "RemoveContainer" containerID="ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f"
May 26 16:05:13.574287 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:05:13.574259    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f\": container with ID starting with ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f not found: ID does not exist" containerID="ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f"
May 26 16:05:13.574389 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.574301    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f"} err="failed to get container status \"ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f\": rpc error: code = NotFound desc = could not find container \"ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f\": container with ID starting with ec1ba41d811a27a2e30c6c5d9e64739049a27a12fd238f20a3a4525d35e9d95f not found: ID does not exist"
May 26 16:05:13.587149 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.587124    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dd78598cd-xk7tp"]
May 26 16:05:13.592367 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.592334    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dd78598cd-xk7tp"]
May 26 16:05:13.711539 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:13.711511    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" path="/var/lib/kubelet/pods/477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53/volumes"
May 26 16:05:14.855086 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.855058    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-b78wx"]
May 26 16:05:14.855460 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.855334    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" containerName="console"
May 26 16:05:14.855460 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.855345    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" containerName="console"
May 26 16:05:14.855460 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.855405    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="477d8ab3-c4c8-4ca9-83d5-0fc8c772ec53" containerName="console"
May 26 16:05:14.858155 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.858138    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:14.860634 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.860613    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 26 16:05:14.868780 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.868761    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b78wx"]
May 26 16:05:14.987494 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.987459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-original-pull-secret\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:14.987626 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.987510    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-kubelet-config\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:14.987626 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:14.987532    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-dbus\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:15.088436 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.088407    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-dbus\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:15.088565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.088478    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-original-pull-secret\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:15.088565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.088509    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-kubelet-config\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:15.088635 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.088584    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-kubelet-config\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:15.088635 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.088593    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-dbus\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:15.090717 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.090687    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5eb3a03-f201-4bb2-9777-e4cdf7db49d0-original-pull-secret\") pod \"global-pull-secret-syncer-b78wx\" (UID: \"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0\") " pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:15.167051 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.166981    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-b78wx"
May 26 16:05:15.290631 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.290597    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-b78wx"]
May 26 16:05:15.294849 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:05:15.294808    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5eb3a03_f201_4bb2_9777_e4cdf7db49d0.slice/crio-3145a5b449a419af900f1a29f28f6c1db6e05a35c06c5937a7d0d5d1f3d14ecf WatchSource:0}: Error finding container 3145a5b449a419af900f1a29f28f6c1db6e05a35c06c5937a7d0d5d1f3d14ecf: Status 404 returned error can't find the container with id 3145a5b449a419af900f1a29f28f6c1db6e05a35c06c5937a7d0d5d1f3d14ecf
May 26 16:05:15.576383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:15.576345    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b78wx" event={"ID":"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0","Type":"ContainerStarted","Data":"3145a5b449a419af900f1a29f28f6c1db6e05a35c06c5937a7d0d5d1f3d14ecf"}
May 26 16:05:19.590237 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:19.590201    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-b78wx" event={"ID":"b5eb3a03-f201-4bb2-9777-e4cdf7db49d0","Type":"ContainerStarted","Data":"3a64abe05c110cebe8fce4ada91fc1224146c55a08900d6d13a43fe967b1cada"}
May 26 16:05:44.852580 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:44.852545    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:44.868002 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:44.867977    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:44.903151 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:44.902946    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-b78wx" podStartSLOduration=26.714811939 podStartE2EDuration="30.902925083s" podCreationTimestamp="2026-05-26 16:05:14 +0000 UTC" firstStartedPulling="2026-05-26 16:05:15.296394036 +0000 UTC m=+262.168652448" lastFinishedPulling="2026-05-26 16:05:19.484507178 +0000 UTC m=+266.356765592" observedRunningTime="2026-05-26 16:05:19.613186564 +0000 UTC m=+266.485444999" watchObservedRunningTime="2026-05-26 16:05:44.902925083 +0000 UTC m=+291.775183518"
May 26 16:05:45.677838 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:45.677813    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 26 16:05:53.594748 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:53.594716    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:05:53.595204 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:53.594727    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:05:53.597600 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:05:53.597580    2561 kubelet.go:1628] "Image garbage collection succeeded"
May 26 16:08:58.185884 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.185850    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-zg4sd"]
May 26 16:08:58.188659 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.188638    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd"
May 26 16:08:58.191175 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.191155    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
May 26 16:08:58.192453 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.192436    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
May 26 16:08:58.192453 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.192444    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-p49g5\""
May 26 16:08:58.198950 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.198930    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-zg4sd"]
May 26 16:08:58.317209 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.317173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx8n6\" (UniqueName: \"kubernetes.io/projected/7fb5b60d-032a-426f-b86d-398b9f7d078f-kube-api-access-mx8n6\") pod \"cert-manager-cainjector-68b757865b-zg4sd\" (UID: \"7fb5b60d-032a-426f-b86d-398b9f7d078f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd"
May 26 16:08:58.317390 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.317289    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fb5b60d-032a-426f-b86d-398b9f7d078f-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-zg4sd\" (UID: \"7fb5b60d-032a-426f-b86d-398b9f7d078f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd"
May 26 16:08:58.418473 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.418433    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mx8n6\" (UniqueName: \"kubernetes.io/projected/7fb5b60d-032a-426f-b86d-398b9f7d078f-kube-api-access-mx8n6\") pod \"cert-manager-cainjector-68b757865b-zg4sd\" (UID: \"7fb5b60d-032a-426f-b86d-398b9f7d078f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd"
May 26 16:08:58.418640 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.418521    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fb5b60d-032a-426f-b86d-398b9f7d078f-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-zg4sd\" (UID: \"7fb5b60d-032a-426f-b86d-398b9f7d078f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd"
May 26 16:08:58.433391 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.433361    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7fb5b60d-032a-426f-b86d-398b9f7d078f-bound-sa-token\") pod \"cert-manager-cainjector-68b757865b-zg4sd\" (UID: \"7fb5b60d-032a-426f-b86d-398b9f7d078f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd"
May 26 16:08:58.433524 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.433463    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx8n6\" (UniqueName: \"kubernetes.io/projected/7fb5b60d-032a-426f-b86d-398b9f7d078f-kube-api-access-mx8n6\") pod \"cert-manager-cainjector-68b757865b-zg4sd\" (UID: \"7fb5b60d-032a-426f-b86d-398b9f7d078f\") " pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd"
May 26 16:08:58.506600 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.506519    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd"
May 26 16:08:58.634680 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.633551    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-68b757865b-zg4sd"]
May 26 16:08:58.640199 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:58.640180    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:08:59.192641 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:08:59.192602    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd" event={"ID":"7fb5b60d-032a-426f-b86d-398b9f7d078f","Type":"ContainerStarted","Data":"8fc99f253ced7f7708a0d0a9e63c220f9bfb6a12a8704d86191d641f99b6b262"}
May 26 16:09:03.207473 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:03.207433    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd" event={"ID":"7fb5b60d-032a-426f-b86d-398b9f7d078f","Type":"ContainerStarted","Data":"7c8beba4823a2224bd39923189b94c94372aff2da186682ee9a7cd4301d9353b"}
May 26 16:09:03.225944 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:03.225874    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-68b757865b-zg4sd" podStartSLOduration=1.684347101 podStartE2EDuration="5.22585933s" podCreationTimestamp="2026-05-26 16:08:58 +0000 UTC" firstStartedPulling="2026-05-26 16:08:58.640326522 +0000 UTC m=+485.512584938" lastFinishedPulling="2026-05-26 16:09:02.181838738 +0000 UTC m=+489.054097167" observedRunningTime="2026-05-26 16:09:03.225023384 +0000 UTC m=+490.097281818" watchObservedRunningTime="2026-05-26 16:09:03.22585933 +0000 UTC m=+490.098117763"
May 26 16:09:13.114408 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.114371    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-79c8d999ff-lqc76"]
May 26 16:09:13.117557 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.117539    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-lqc76"
May 26 16:09:13.120105 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.120082    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-cvgdd\""
May 26 16:09:13.124687 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.124667    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-lqc76"]
May 26 16:09:13.241774 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.241742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7b62c2-b023-4e98-a1fb-34b91ef1426b-bound-sa-token\") pod \"cert-manager-79c8d999ff-lqc76\" (UID: \"ef7b62c2-b023-4e98-a1fb-34b91ef1426b\") " pod="cert-manager/cert-manager-79c8d999ff-lqc76"
May 26 16:09:13.241953 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.241802    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7rc5\" (UniqueName: \"kubernetes.io/projected/ef7b62c2-b023-4e98-a1fb-34b91ef1426b-kube-api-access-d7rc5\") pod \"cert-manager-79c8d999ff-lqc76\" (UID: \"ef7b62c2-b023-4e98-a1fb-34b91ef1426b\") " pod="cert-manager/cert-manager-79c8d999ff-lqc76"
May 26 16:09:13.342737 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.342708    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d7rc5\" (UniqueName: \"kubernetes.io/projected/ef7b62c2-b023-4e98-a1fb-34b91ef1426b-kube-api-access-d7rc5\") pod \"cert-manager-79c8d999ff-lqc76\" (UID: \"ef7b62c2-b023-4e98-a1fb-34b91ef1426b\") " pod="cert-manager/cert-manager-79c8d999ff-lqc76"
May 26 16:09:13.342836 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.342777    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7b62c2-b023-4e98-a1fb-34b91ef1426b-bound-sa-token\") pod \"cert-manager-79c8d999ff-lqc76\" (UID: \"ef7b62c2-b023-4e98-a1fb-34b91ef1426b\") " pod="cert-manager/cert-manager-79c8d999ff-lqc76"
May 26 16:09:13.350916 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.350869    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef7b62c2-b023-4e98-a1fb-34b91ef1426b-bound-sa-token\") pod \"cert-manager-79c8d999ff-lqc76\" (UID: \"ef7b62c2-b023-4e98-a1fb-34b91ef1426b\") " pod="cert-manager/cert-manager-79c8d999ff-lqc76"
May 26 16:09:13.351091 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.350990    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7rc5\" (UniqueName: \"kubernetes.io/projected/ef7b62c2-b023-4e98-a1fb-34b91ef1426b-kube-api-access-d7rc5\") pod \"cert-manager-79c8d999ff-lqc76\" (UID: \"ef7b62c2-b023-4e98-a1fb-34b91ef1426b\") " pod="cert-manager/cert-manager-79c8d999ff-lqc76"
May 26 16:09:13.427492 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.427424    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-79c8d999ff-lqc76"
May 26 16:09:13.551634 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:13.551602    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-79c8d999ff-lqc76"]
May 26 16:09:13.554765 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:09:13.554734    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef7b62c2_b023_4e98_a1fb_34b91ef1426b.slice/crio-fcf63eea62d03dd8edad1510b9482086e68bc6170e180480b865b584ac243cac WatchSource:0}: Error finding container fcf63eea62d03dd8edad1510b9482086e68bc6170e180480b865b584ac243cac: Status 404 returned error can't find the container with id fcf63eea62d03dd8edad1510b9482086e68bc6170e180480b865b584ac243cac
May 26 16:09:14.239299 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:14.239265    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-lqc76" event={"ID":"ef7b62c2-b023-4e98-a1fb-34b91ef1426b","Type":"ContainerStarted","Data":"0a8f81cd03b140683d1d6a9c6140cd1b689b00ab212f7787bc0dbee7fbe32992"}
May 26 16:09:14.239299 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:14.239301    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-79c8d999ff-lqc76" event={"ID":"ef7b62c2-b023-4e98-a1fb-34b91ef1426b","Type":"ContainerStarted","Data":"fcf63eea62d03dd8edad1510b9482086e68bc6170e180480b865b584ac243cac"}
May 26 16:09:14.263814 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:14.263765    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="cert-manager/cert-manager-79c8d999ff-lqc76" podStartSLOduration=1.2637499079999999 podStartE2EDuration="1.263749908s" podCreationTimestamp="2026-05-26 16:09:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:09:14.26232574 +0000 UTC m=+501.134584174" watchObservedRunningTime="2026-05-26 16:09:14.263749908 +0000 UTC m=+501.136008343"
May 26 16:09:49.828856 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.828767    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"]
May 26 16:09:49.831135 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.831119    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:49.838723 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.838702    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
May 26 16:09:49.839608 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.839588    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dvkwk\""
May 26 16:09:49.839717 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.839682    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
May 26 16:09:49.840596 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.840580    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
May 26 16:09:49.840795 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.840781    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
May 26 16:09:49.843285 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.843269    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
May 26 16:09:49.850938 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.850917    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"]
May 26 16:09:49.947565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.947527    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f032dba2-6b99-4f76-8e90-45f2a1c45db3-metrics-cert\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:49.947565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.947569    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f032dba2-6b99-4f76-8e90-45f2a1c45db3-cert\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:49.947791 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.947600    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7bw\" (UniqueName: \"kubernetes.io/projected/f032dba2-6b99-4f76-8e90-45f2a1c45db3-kube-api-access-4x7bw\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:49.947791 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:49.947672    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f032dba2-6b99-4f76-8e90-45f2a1c45db3-manager-config\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.049061 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.049030    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7bw\" (UniqueName: \"kubernetes.io/projected/f032dba2-6b99-4f76-8e90-45f2a1c45db3-kube-api-access-4x7bw\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.049268 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.049079    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f032dba2-6b99-4f76-8e90-45f2a1c45db3-manager-config\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.049268 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.049161    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f032dba2-6b99-4f76-8e90-45f2a1c45db3-metrics-cert\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.049268 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.049196    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f032dba2-6b99-4f76-8e90-45f2a1c45db3-cert\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.049843 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.049819    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f032dba2-6b99-4f76-8e90-45f2a1c45db3-manager-config\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.051712 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.051692    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f032dba2-6b99-4f76-8e90-45f2a1c45db3-cert\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.051799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.051749    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f032dba2-6b99-4f76-8e90-45f2a1c45db3-metrics-cert\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.059947 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.059920    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7bw\" (UniqueName: \"kubernetes.io/projected/f032dba2-6b99-4f76-8e90-45f2a1c45db3-kube-api-access-4x7bw\") pod \"lws-controller-manager-669cc6566-4vh99\" (UID: \"f032dba2-6b99-4f76-8e90-45f2a1c45db3\") " pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.141242 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.141158    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:50.271916 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.271866    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"]
May 26 16:09:50.276252 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:09:50.276226    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf032dba2_6b99_4f76_8e90_45f2a1c45db3.slice/crio-662a972eac24343350d5de92af3624f542a145d39ad78c2bcc6c215f078aae7e WatchSource:0}: Error finding container 662a972eac24343350d5de92af3624f542a145d39ad78c2bcc6c215f078aae7e: Status 404 returned error can't find the container with id 662a972eac24343350d5de92af3624f542a145d39ad78c2bcc6c215f078aae7e
May 26 16:09:50.344572 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:50.344536    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99" event={"ID":"f032dba2-6b99-4f76-8e90-45f2a1c45db3","Type":"ContainerStarted","Data":"662a972eac24343350d5de92af3624f542a145d39ad78c2bcc6c215f078aae7e"}
May 26 16:09:53.355494 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:53.355457    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99" event={"ID":"f032dba2-6b99-4f76-8e90-45f2a1c45db3","Type":"ContainerStarted","Data":"e23d217d1ce4a1ee60c0388b81137cf68630c0c9af47f795f378be42372c7e11"}
May 26 16:09:53.355884 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:53.355596    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:09:53.373872 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:09:53.373815    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99" podStartSLOduration=1.739117631 podStartE2EDuration="4.373799596s" podCreationTimestamp="2026-05-26 16:09:49 +0000 UTC" firstStartedPulling="2026-05-26 16:09:50.278380961 +0000 UTC m=+537.150639383" lastFinishedPulling="2026-05-26 16:09:52.913062933 +0000 UTC m=+539.785321348" observedRunningTime="2026-05-26 16:09:53.372280398 +0000 UTC m=+540.244538832" watchObservedRunningTime="2026-05-26 16:09:53.373799596 +0000 UTC m=+540.246058029"
May 26 16:10:04.361317 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:04.361287    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-669cc6566-4vh99"
May 26 16:10:22.496571 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.496536    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"]
May 26 16:10:22.500373 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.500352    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:22.503060 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.503039    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-frtrk\""
May 26 16:10:22.503214 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.503115    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
May 26 16:10:22.503377 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.503360    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
May 26 16:10:22.510696 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.510664    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"]
May 26 16:10:22.527915 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.527879    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s5rg\" (UniqueName: \"kubernetes.io/projected/ca33e181-a488-43f6-ac69-b8043f9024f4-kube-api-access-2s5rg\") pod \"kuadrant-operator-controller-manager-57dd85bb4-pjll2\" (UID: \"ca33e181-a488-43f6-ac69-b8043f9024f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:22.528016 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.527937    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca33e181-a488-43f6-ac69-b8043f9024f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-pjll2\" (UID: \"ca33e181-a488-43f6-ac69-b8043f9024f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:22.629386 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.629358    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2s5rg\" (UniqueName: \"kubernetes.io/projected/ca33e181-a488-43f6-ac69-b8043f9024f4-kube-api-access-2s5rg\") pod \"kuadrant-operator-controller-manager-57dd85bb4-pjll2\" (UID: \"ca33e181-a488-43f6-ac69-b8043f9024f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:22.629544 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.629399    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca33e181-a488-43f6-ac69-b8043f9024f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-pjll2\" (UID: \"ca33e181-a488-43f6-ac69-b8043f9024f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:22.629724 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.629709    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca33e181-a488-43f6-ac69-b8043f9024f4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57dd85bb4-pjll2\" (UID: \"ca33e181-a488-43f6-ac69-b8043f9024f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:22.658248 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.658218    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s5rg\" (UniqueName: \"kubernetes.io/projected/ca33e181-a488-43f6-ac69-b8043f9024f4-kube-api-access-2s5rg\") pod \"kuadrant-operator-controller-manager-57dd85bb4-pjll2\" (UID: \"ca33e181-a488-43f6-ac69-b8043f9024f4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:22.811467 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.811439    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:22.946091 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:22.946062    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"]
May 26 16:10:22.949381 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:10:22.949345    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca33e181_a488_43f6_ac69_b8043f9024f4.slice/crio-f2177990f1a3a5afeacfa407c0a908c838bc5963120b044459f8529c66175388 WatchSource:0}: Error finding container f2177990f1a3a5afeacfa407c0a908c838bc5963120b044459f8529c66175388: Status 404 returned error can't find the container with id f2177990f1a3a5afeacfa407c0a908c838bc5963120b044459f8529c66175388
May 26 16:10:23.446713 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:23.446677    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2" event={"ID":"ca33e181-a488-43f6-ac69-b8043f9024f4","Type":"ContainerStarted","Data":"f2177990f1a3a5afeacfa407c0a908c838bc5963120b044459f8529c66175388"}
May 26 16:10:27.465640 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:27.465604    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2" event={"ID":"ca33e181-a488-43f6-ac69-b8043f9024f4","Type":"ContainerStarted","Data":"187ddd53ea684a77a65b180e964fd86a9abdf5881e3da572f9f765e71aa22d46"}
May 26 16:10:27.466155 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:27.465760    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:27.491312 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:27.491254    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2" podStartSLOduration=1.706868315 podStartE2EDuration="5.491237748s" podCreationTimestamp="2026-05-26 16:10:22 +0000 UTC" firstStartedPulling="2026-05-26 16:10:22.952396187 +0000 UTC m=+569.824654599" lastFinishedPulling="2026-05-26 16:10:26.736765607 +0000 UTC m=+573.609024032" observedRunningTime="2026-05-26 16:10:27.490534103 +0000 UTC m=+574.362792536" watchObservedRunningTime="2026-05-26 16:10:27.491237748 +0000 UTC m=+574.363496182"
May 26 16:10:38.471588 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:38.471555    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57dd85bb4-pjll2"
May 26 16:10:53.617751 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:53.617723    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:10:53.618840 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:10:53.618820    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:11:14.885465 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:14.885384    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-674b59b84c-wxtvv"]
May 26 16:11:14.887610 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:14.887589    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-wxtvv"
May 26 16:11:14.890515 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:14.890493    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w6sb5\""
May 26 16:11:14.895027 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:14.894998    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-wxtvv"]
May 26 16:11:15.065464 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:15.065431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrk5\" (UniqueName: \"kubernetes.io/projected/427cdc13-8c54-488f-ba4e-09cc66c04b6a-kube-api-access-pmrk5\") pod \"authorino-674b59b84c-wxtvv\" (UID: \"427cdc13-8c54-488f-ba4e-09cc66c04b6a\") " pod="kuadrant-system/authorino-674b59b84c-wxtvv"
May 26 16:11:15.166216 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:15.166130    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrk5\" (UniqueName: \"kubernetes.io/projected/427cdc13-8c54-488f-ba4e-09cc66c04b6a-kube-api-access-pmrk5\") pod \"authorino-674b59b84c-wxtvv\" (UID: \"427cdc13-8c54-488f-ba4e-09cc66c04b6a\") " pod="kuadrant-system/authorino-674b59b84c-wxtvv"
May 26 16:11:15.175476 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:15.175452    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrk5\" (UniqueName: \"kubernetes.io/projected/427cdc13-8c54-488f-ba4e-09cc66c04b6a-kube-api-access-pmrk5\") pod \"authorino-674b59b84c-wxtvv\" (UID: \"427cdc13-8c54-488f-ba4e-09cc66c04b6a\") " pod="kuadrant-system/authorino-674b59b84c-wxtvv"
May 26 16:11:15.198369 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:15.198348    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-wxtvv"
May 26 16:11:15.333609 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:15.333580    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-674b59b84c-wxtvv"]
May 26 16:11:15.334153 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:11:15.334133    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod427cdc13_8c54_488f_ba4e_09cc66c04b6a.slice/crio-61ec96b780b8b31ad68db0d5d68b9d8b94cc39b5ad668e50aa0ed9c078fe6ead WatchSource:0}: Error finding container 61ec96b780b8b31ad68db0d5d68b9d8b94cc39b5ad668e50aa0ed9c078fe6ead: Status 404 returned error can't find the container with id 61ec96b780b8b31ad68db0d5d68b9d8b94cc39b5ad668e50aa0ed9c078fe6ead
May 26 16:11:15.627418 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:15.627388    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-wxtvv" event={"ID":"427cdc13-8c54-488f-ba4e-09cc66c04b6a","Type":"ContainerStarted","Data":"61ec96b780b8b31ad68db0d5d68b9d8b94cc39b5ad668e50aa0ed9c078fe6ead"}
May 26 16:11:17.635730 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:17.635691    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-wxtvv" event={"ID":"427cdc13-8c54-488f-ba4e-09cc66c04b6a","Type":"ContainerStarted","Data":"f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce"}
May 26 16:11:17.655609 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:17.655558    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-674b59b84c-wxtvv" podStartSLOduration=2.147420674 podStartE2EDuration="3.655541191s" podCreationTimestamp="2026-05-26 16:11:14 +0000 UTC" firstStartedPulling="2026-05-26 16:11:15.33581152 +0000 UTC m=+622.208069933" lastFinishedPulling="2026-05-26 16:11:16.843932035 +0000 UTC m=+623.716190450" observedRunningTime="2026-05-26 16:11:17.65432794 +0000 UTC m=+624.526586374" watchObservedRunningTime="2026-05-26 16:11:17.655541191 +0000 UTC m=+624.527799626"
May 26 16:11:18.297704 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:18.297669    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-wxtvv"]
May 26 16:11:19.641384 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:19.641319    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kuadrant-system/authorino-674b59b84c-wxtvv" podUID="427cdc13-8c54-488f-ba4e-09cc66c04b6a" containerName="authorino" containerID="cri-o://f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce" gracePeriod=30
May 26 16:11:19.886992 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:19.886971    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-wxtvv"
May 26 16:11:19.904142 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:19.904071    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmrk5\" (UniqueName: \"kubernetes.io/projected/427cdc13-8c54-488f-ba4e-09cc66c04b6a-kube-api-access-pmrk5\") pod \"427cdc13-8c54-488f-ba4e-09cc66c04b6a\" (UID: \"427cdc13-8c54-488f-ba4e-09cc66c04b6a\") "
May 26 16:11:19.906151 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:19.906128    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427cdc13-8c54-488f-ba4e-09cc66c04b6a-kube-api-access-pmrk5" (OuterVolumeSpecName: "kube-api-access-pmrk5") pod "427cdc13-8c54-488f-ba4e-09cc66c04b6a" (UID: "427cdc13-8c54-488f-ba4e-09cc66c04b6a"). InnerVolumeSpecName "kube-api-access-pmrk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:11:20.004629 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.004589    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrk5\" (UniqueName: \"kubernetes.io/projected/427cdc13-8c54-488f-ba4e-09cc66c04b6a-kube-api-access-pmrk5\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:11:20.645622 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.645587    2561 generic.go:358] "Generic (PLEG): container finished" podID="427cdc13-8c54-488f-ba4e-09cc66c04b6a" containerID="f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce" exitCode=0
May 26 16:11:20.646145 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.645634    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-674b59b84c-wxtvv"
May 26 16:11:20.646145 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.645667    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-wxtvv" event={"ID":"427cdc13-8c54-488f-ba4e-09cc66c04b6a","Type":"ContainerDied","Data":"f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce"}
May 26 16:11:20.646145 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.645710    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-674b59b84c-wxtvv" event={"ID":"427cdc13-8c54-488f-ba4e-09cc66c04b6a","Type":"ContainerDied","Data":"61ec96b780b8b31ad68db0d5d68b9d8b94cc39b5ad668e50aa0ed9c078fe6ead"}
May 26 16:11:20.646145 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.645730    2561 scope.go:117] "RemoveContainer" containerID="f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce"
May 26 16:11:20.656736 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.656717    2561 scope.go:117] "RemoveContainer" containerID="f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce"
May 26 16:11:20.657054 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:11:20.657034    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce\": container with ID starting with f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce not found: ID does not exist" containerID="f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce"
May 26 16:11:20.657133 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.657066    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce"} err="failed to get container status \"f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce\": rpc error: code = NotFound desc = could not find container \"f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce\": container with ID starting with f74be86d56ee6aa591a1fe191e1fc2743e472f3923f6e4280a3a5552b563d7ce not found: ID does not exist"
May 26 16:11:20.669115 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.669086    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-674b59b84c-wxtvv"]
May 26 16:11:20.672721 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:20.672693    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-674b59b84c-wxtvv"]
May 26 16:11:21.711365 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:21.711331    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="427cdc13-8c54-488f-ba4e-09cc66c04b6a" path="/var/lib/kubelet/pods/427cdc13-8c54-488f-ba4e-09cc66c04b6a/volumes"
May 26 16:11:37.338856 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.338816    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68bd676465-488m8"]
May 26 16:11:37.341946 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.339127    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="427cdc13-8c54-488f-ba4e-09cc66c04b6a" containerName="authorino"
May 26 16:11:37.341946 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.339138    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="427cdc13-8c54-488f-ba4e-09cc66c04b6a" containerName="authorino"
May 26 16:11:37.341946 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.339196    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="427cdc13-8c54-488f-ba4e-09cc66c04b6a" containerName="authorino"
May 26 16:11:37.343106 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.343091    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-488m8"
May 26 16:11:37.347469 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.347440    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
May 26 16:11:37.349019 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.348994    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-488m8"]
May 26 16:11:37.350167 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.350148    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-w6sb5\""
May 26 16:11:37.450072 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.450038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtjb\" (UniqueName: \"kubernetes.io/projected/ee83b144-e776-4023-a729-91d0f4292b86-kube-api-access-crtjb\") pod \"authorino-68bd676465-488m8\" (UID: \"ee83b144-e776-4023-a729-91d0f4292b86\") " pod="kuadrant-system/authorino-68bd676465-488m8"
May 26 16:11:37.450228 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.450108    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee83b144-e776-4023-a729-91d0f4292b86-tls-cert\") pod \"authorino-68bd676465-488m8\" (UID: \"ee83b144-e776-4023-a729-91d0f4292b86\") " pod="kuadrant-system/authorino-68bd676465-488m8"
May 26 16:11:37.551452 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.551418    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee83b144-e776-4023-a729-91d0f4292b86-tls-cert\") pod \"authorino-68bd676465-488m8\" (UID: \"ee83b144-e776-4023-a729-91d0f4292b86\") " pod="kuadrant-system/authorino-68bd676465-488m8"
May 26 16:11:37.551605 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.551476    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-crtjb\" (UniqueName: \"kubernetes.io/projected/ee83b144-e776-4023-a729-91d0f4292b86-kube-api-access-crtjb\") pod \"authorino-68bd676465-488m8\" (UID: \"ee83b144-e776-4023-a729-91d0f4292b86\") " pod="kuadrant-system/authorino-68bd676465-488m8"
May 26 16:11:37.554020 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.553991    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ee83b144-e776-4023-a729-91d0f4292b86-tls-cert\") pod \"authorino-68bd676465-488m8\" (UID: \"ee83b144-e776-4023-a729-91d0f4292b86\") " pod="kuadrant-system/authorino-68bd676465-488m8"
May 26 16:11:37.563526 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.563500    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtjb\" (UniqueName: \"kubernetes.io/projected/ee83b144-e776-4023-a729-91d0f4292b86-kube-api-access-crtjb\") pod \"authorino-68bd676465-488m8\" (UID: \"ee83b144-e776-4023-a729-91d0f4292b86\") " pod="kuadrant-system/authorino-68bd676465-488m8"
May 26 16:11:37.653201 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.653131    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68bd676465-488m8"
May 26 16:11:37.794423 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:37.794394    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68bd676465-488m8"]
May 26 16:11:37.797469 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:11:37.797441    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee83b144_e776_4023_a729_91d0f4292b86.slice/crio-4973e08ad1b2aa449642cf449f1bf60179b80aa81f8e3cb1cfe450c22299b503 WatchSource:0}: Error finding container 4973e08ad1b2aa449642cf449f1bf60179b80aa81f8e3cb1cfe450c22299b503: Status 404 returned error can't find the container with id 4973e08ad1b2aa449642cf449f1bf60179b80aa81f8e3cb1cfe450c22299b503
May 26 16:11:38.704958 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:38.704920    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-488m8" event={"ID":"ee83b144-e776-4023-a729-91d0f4292b86","Type":"ContainerStarted","Data":"4973e08ad1b2aa449642cf449f1bf60179b80aa81f8e3cb1cfe450c22299b503"}
May 26 16:11:39.711970 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:39.711936    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68bd676465-488m8" event={"ID":"ee83b144-e776-4023-a729-91d0f4292b86","Type":"ContainerStarted","Data":"34b7c6a8bba443b3e3075a6e9bf52cd2cbf0600f1216049ff2c7675fb9c7f850"}
May 26 16:11:39.730725 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:11:39.730681    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kuadrant-system/authorino-68bd676465-488m8" podStartSLOduration=1.811741981 podStartE2EDuration="2.730669157s" podCreationTimestamp="2026-05-26 16:11:37 +0000 UTC" firstStartedPulling="2026-05-26 16:11:37.798657847 +0000 UTC m=+644.670916259" lastFinishedPulling="2026-05-26 16:11:38.717585023 +0000 UTC m=+645.589843435" observedRunningTime="2026-05-26 16:11:39.728828434 +0000 UTC m=+646.601086867" watchObservedRunningTime="2026-05-26 16:11:39.730669157 +0000 UTC m=+646.602927591"
May 26 16:13:17.511687 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.511650    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-s2297"]
May 26 16:13:17.514902 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.514867    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:17.517474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.517453    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 26 16:13:17.517641 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.517479    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 26 16:13:17.518274 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.518258    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-sl2mc\""
May 26 16:13:17.518363 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.518285    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 26 16:13:17.524447 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.524419    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-s2297"]
May 26 16:13:17.617152 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.617119    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc2vs\" (UniqueName: \"kubernetes.io/projected/5218fd50-ee10-456d-aa49-f79111f8422b-kube-api-access-cc2vs\") pod \"odh-model-controller-696fc77849-s2297\" (UID: \"5218fd50-ee10-456d-aa49-f79111f8422b\") " pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:17.617313 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.617161    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5218fd50-ee10-456d-aa49-f79111f8422b-cert\") pod \"odh-model-controller-696fc77849-s2297\" (UID: \"5218fd50-ee10-456d-aa49-f79111f8422b\") " pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:17.718223 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.718195    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cc2vs\" (UniqueName: \"kubernetes.io/projected/5218fd50-ee10-456d-aa49-f79111f8422b-kube-api-access-cc2vs\") pod \"odh-model-controller-696fc77849-s2297\" (UID: \"5218fd50-ee10-456d-aa49-f79111f8422b\") " pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:17.718223 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.718229    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5218fd50-ee10-456d-aa49-f79111f8422b-cert\") pod \"odh-model-controller-696fc77849-s2297\" (UID: \"5218fd50-ee10-456d-aa49-f79111f8422b\") " pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:17.720717 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.720689    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5218fd50-ee10-456d-aa49-f79111f8422b-cert\") pod \"odh-model-controller-696fc77849-s2297\" (UID: \"5218fd50-ee10-456d-aa49-f79111f8422b\") " pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:17.726597 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.726574    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc2vs\" (UniqueName: \"kubernetes.io/projected/5218fd50-ee10-456d-aa49-f79111f8422b-kube-api-access-cc2vs\") pod \"odh-model-controller-696fc77849-s2297\" (UID: \"5218fd50-ee10-456d-aa49-f79111f8422b\") " pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:17.827346 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.827311    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:17.965823 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:17.965751    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-s2297"]
May 26 16:13:17.968515 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:13:17.968488    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5218fd50_ee10_456d_aa49_f79111f8422b.slice/crio-8c666e635eb720cce2765f57e521a610e7625f789c05116e07f145fea7e276ae WatchSource:0}: Error finding container 8c666e635eb720cce2765f57e521a610e7625f789c05116e07f145fea7e276ae: Status 404 returned error can't find the container with id 8c666e635eb720cce2765f57e521a610e7625f789c05116e07f145fea7e276ae
May 26 16:13:18.023927 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:18.023883    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-s2297" event={"ID":"5218fd50-ee10-456d-aa49-f79111f8422b","Type":"ContainerStarted","Data":"8c666e635eb720cce2765f57e521a610e7625f789c05116e07f145fea7e276ae"}
May 26 16:13:21.036051 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:21.036014    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-s2297" event={"ID":"5218fd50-ee10-456d-aa49-f79111f8422b","Type":"ContainerStarted","Data":"958a2ea23217bccdbb5de1606cd2cf2c47e5245727f68cc705a271cd8ac04f7e"}
May 26 16:13:21.036429 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:21.036136    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:13:21.054994 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:21.054939    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-s2297" podStartSLOduration=1.162190078 podStartE2EDuration="4.054924934s" podCreationTimestamp="2026-05-26 16:13:17 +0000 UTC" firstStartedPulling="2026-05-26 16:13:17.969833132 +0000 UTC m=+744.842091544" lastFinishedPulling="2026-05-26 16:13:20.862567986 +0000 UTC m=+747.734826400" observedRunningTime="2026-05-26 16:13:21.053095861 +0000 UTC m=+747.925354296" watchObservedRunningTime="2026-05-26 16:13:21.054924934 +0000 UTC m=+747.927183368"
May 26 16:13:32.042006 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:13:32.041976    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-s2297"
May 26 16:14:09.678045 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.677963    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"]
May 26 16:14:09.688391 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.688359    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.691998 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.691974    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:14:09.692225 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.692036    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:14:09.692562 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.692093    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:14:09.692854 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.692142    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
May 26 16:14:09.695695 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.695671    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"]
May 26 16:14:09.772884 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.772854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.773089 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.772915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpv28\" (UniqueName: \"kubernetes.io/projected/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kube-api-access-fpv28\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.773089 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.772983    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-home\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.773089 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.773022    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.773089 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.773070    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-model-cache\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.773308 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.773173    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-dshm\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.773308 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.773211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tls-certs\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.874637 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.874609    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-dshm\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.874637 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.874640    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tls-certs\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.874920 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.874665    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.874920 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.874690    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fpv28\" (UniqueName: \"kubernetes.io/projected/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kube-api-access-fpv28\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.874920 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.874731    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-home\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.874920 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.874766    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.874920 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.874833    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-model-cache\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.875212 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.875137    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kserve-provision-location\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.875471 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.875446    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tmp-dir\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.875553 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.875452    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-model-cache\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.875553 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.875513    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-home\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.876956 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.876934    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-dshm\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.877140 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.877123    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tls-certs\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:09.885152 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:09.885128    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpv28\" (UniqueName: \"kubernetes.io/projected/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kube-api-access-fpv28\") pod \"scheduler-inline-config-test-kserve-c9cfc44d9-4crk8\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:10.000881 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:10.000782    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:10.139775 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:10.139749    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"]
May 26 16:14:10.142255 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:14:10.142224    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f4fb3d_b9ee_4e62_8160_2b35918f720f.slice/crio-7181611f2bca621cae9e37daa60cbee1d2c8d793c66507dbe437230b7cb8212c WatchSource:0}: Error finding container 7181611f2bca621cae9e37daa60cbee1d2c8d793c66507dbe437230b7cb8212c: Status 404 returned error can't find the container with id 7181611f2bca621cae9e37daa60cbee1d2c8d793c66507dbe437230b7cb8212c
May 26 16:14:10.144072 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:10.144053    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:14:10.202679 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:10.202645    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8" event={"ID":"33f4fb3d-b9ee-4e62-8160-2b35918f720f","Type":"ContainerStarted","Data":"7181611f2bca621cae9e37daa60cbee1d2c8d793c66507dbe437230b7cb8212c"}
May 26 16:14:14.220983 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:14.220943    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8" event={"ID":"33f4fb3d-b9ee-4e62-8160-2b35918f720f","Type":"ContainerStarted","Data":"f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d"}
May 26 16:14:19.237816 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:19.237780    2561 generic.go:358] "Generic (PLEG): container finished" podID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" containerID="f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d" exitCode=0
May 26 16:14:19.238223 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:19.237857    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8" event={"ID":"33f4fb3d-b9ee-4e62-8160-2b35918f720f","Type":"ContainerDied","Data":"f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d"}
May 26 16:14:21.248104 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:21.248067    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8" event={"ID":"33f4fb3d-b9ee-4e62-8160-2b35918f720f","Type":"ContainerStarted","Data":"96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204"}
May 26 16:14:21.269309 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:21.269256    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8" podStartSLOduration=2.009229163 podStartE2EDuration="12.269239041s" podCreationTimestamp="2026-05-26 16:14:09 +0000 UTC" firstStartedPulling="2026-05-26 16:14:10.144191483 +0000 UTC m=+797.016449898" lastFinishedPulling="2026-05-26 16:14:20.404201362 +0000 UTC m=+807.276459776" observedRunningTime="2026-05-26 16:14:21.268182968 +0000 UTC m=+808.140441401" watchObservedRunningTime="2026-05-26 16:14:21.269239041 +0000 UTC m=+808.141497477"
May 26 16:14:30.001227 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:30.001187    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:30.001227 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:30.001232    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:30.014502 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:30.014474    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:14:30.288322 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:14:30.288292    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:15:36.911476 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:36.911426    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"]
May 26 16:15:36.912083 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:36.911860    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8" podUID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" containerName="main" containerID="cri-o://96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204" gracePeriod=30
May 26 16:15:37.161288 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.161266    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:15:37.242784 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.242701    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fpv28\" (UniqueName: \"kubernetes.io/projected/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kube-api-access-fpv28\") pod \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") "
May 26 16:15:37.242784 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.242746    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-dshm\") pod \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") "
May 26 16:15:37.243019 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.242825    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tls-certs\") pod \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") "
May 26 16:15:37.243019 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.242855    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-home\") pod \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") "
May 26 16:15:37.243019 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.242902    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kserve-provision-location\") pod \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") "
May 26 16:15:37.243019 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.242935    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-model-cache\") pod \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") "
May 26 16:15:37.243019 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.242964    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tmp-dir\") pod \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\" (UID: \"33f4fb3d-b9ee-4e62-8160-2b35918f720f\") "
May 26 16:15:37.243242 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.243144    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-home" (OuterVolumeSpecName: "home") pod "33f4fb3d-b9ee-4e62-8160-2b35918f720f" (UID: "33f4fb3d-b9ee-4e62-8160-2b35918f720f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:37.243389 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.243345    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33f4fb3d-b9ee-4e62-8160-2b35918f720f" (UID: "33f4fb3d-b9ee-4e62-8160-2b35918f720f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:37.243445 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.243383    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-model-cache" (OuterVolumeSpecName: "model-cache") pod "33f4fb3d-b9ee-4e62-8160-2b35918f720f" (UID: "33f4fb3d-b9ee-4e62-8160-2b35918f720f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:37.245512 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.245478    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-dshm" (OuterVolumeSpecName: "dshm") pod "33f4fb3d-b9ee-4e62-8160-2b35918f720f" (UID: "33f4fb3d-b9ee-4e62-8160-2b35918f720f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:37.245512 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.245491    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33f4fb3d-b9ee-4e62-8160-2b35918f720f" (UID: "33f4fb3d-b9ee-4e62-8160-2b35918f720f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:15:37.245649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.245565    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kube-api-access-fpv28" (OuterVolumeSpecName: "kube-api-access-fpv28") pod "33f4fb3d-b9ee-4e62-8160-2b35918f720f" (UID: "33f4fb3d-b9ee-4e62-8160-2b35918f720f"). InnerVolumeSpecName "kube-api-access-fpv28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:15:37.299210 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.299165    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33f4fb3d-b9ee-4e62-8160-2b35918f720f" (UID: "33f4fb3d-b9ee-4e62-8160-2b35918f720f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:15:37.344423 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.344392    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:15:37.344423 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.344418    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:15:37.344592 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.344428    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpv28\" (UniqueName: \"kubernetes.io/projected/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kube-api-access-fpv28\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:15:37.344592 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.344448    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:15:37.344592 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.344458    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f4fb3d-b9ee-4e62-8160-2b35918f720f-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:15:37.344592 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.344466    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:15:37.344592 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.344474    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33f4fb3d-b9ee-4e62-8160-2b35918f720f-kserve-provision-location\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:15:37.507222 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.507135    2561 generic.go:358] "Generic (PLEG): container finished" podID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" containerID="96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204" exitCode=0
May 26 16:15:37.507383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.507226    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"
May 26 16:15:37.507383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.507226    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8" event={"ID":"33f4fb3d-b9ee-4e62-8160-2b35918f720f","Type":"ContainerDied","Data":"96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204"}
May 26 16:15:37.507383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.507270    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8" event={"ID":"33f4fb3d-b9ee-4e62-8160-2b35918f720f","Type":"ContainerDied","Data":"7181611f2bca621cae9e37daa60cbee1d2c8d793c66507dbe437230b7cb8212c"}
May 26 16:15:37.507383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.507290    2561 scope.go:117] "RemoveContainer" containerID="96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204"
May 26 16:15:37.515612 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.515591    2561 scope.go:117] "RemoveContainer" containerID="f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d"
May 26 16:15:37.535736 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.535712    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"]
May 26 16:15:37.539156 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.539127    2561 scope.go:117] "RemoveContainer" containerID="96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204"
May 26 16:15:37.539461 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:15:37.539440    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204\": container with ID starting with 96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204 not found: ID does not exist" containerID="96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204"
May 26 16:15:37.539554 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.539469    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204"} err="failed to get container status \"96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204\": rpc error: code = NotFound desc = could not find container \"96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204\": container with ID starting with 96bef2648983b5be5bad5fe292ac314437ab2287c6dc49e5aeffc8de5a072204 not found: ID does not exist"
May 26 16:15:37.539554 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.539487    2561 scope.go:117] "RemoveContainer" containerID="f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d"
May 26 16:15:37.539734 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:15:37.539719    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d\": container with ID starting with f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d not found: ID does not exist" containerID="f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d"
May 26 16:15:37.539780 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.539737    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d"} err="failed to get container status \"f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d\": rpc error: code = NotFound desc = could not find container \"f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d\": container with ID starting with f206307e17e2d2108dd23666cffacb53b0d85af82ad1bbd326df69e4a556098d not found: ID does not exist"
May 26 16:15:37.539959 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.539942    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-c9cfc44d9-4crk8"]
May 26 16:15:37.711781 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:37.711747    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" path="/var/lib/kubelet/pods/33f4fb3d-b9ee-4e62-8160-2b35918f720f/volumes"
May 26 16:15:44.036608 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.036529    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"]
May 26 16:15:44.036974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.036843    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" containerName="storage-initializer"
May 26 16:15:44.036974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.036854    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" containerName="storage-initializer"
May 26 16:15:44.036974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.036869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" containerName="main"
May 26 16:15:44.036974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.036876    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" containerName="main"
May 26 16:15:44.036974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.036946    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="33f4fb3d-b9ee-4e62-8160-2b35918f720f" containerName="main"
May 26 16:15:44.038839 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.038823    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.044430 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.044408    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:15:44.045701 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.045680    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:15:44.045884 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.045866    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
May 26 16:15:44.046773 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.046756    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:15:44.069147 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.069122    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"]
May 26 16:15:44.100298 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.100271    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638600b9-bf0b-4b64-abae-46fe978d90d8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.100549 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.100494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/638600b9-bf0b-4b64-abae-46fe978d90d8-kube-api-access-vw4z6\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.100649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.100561    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-dshm\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.100649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.100617    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.100735 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.100659    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-model-cache\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.100735 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.100698    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-home\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.100806 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.100765    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201289 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201257    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638600b9-bf0b-4b64-abae-46fe978d90d8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201437 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201296    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/638600b9-bf0b-4b64-abae-46fe978d90d8-kube-api-access-vw4z6\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201437 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201324    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-dshm\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201437 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201369    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201437 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201404    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-model-cache\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201437 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201437    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-home\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201686 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201497    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201882 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201854    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-model-cache\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.201882 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201872    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-home\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.202060 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.201964    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.202060 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.202018    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-kserve-provision-location\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.203707 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.203679    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-dshm\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.203916 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.203881    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638600b9-bf0b-4b64-abae-46fe978d90d8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.210564 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.210542    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/638600b9-bf0b-4b64-abae-46fe978d90d8-kube-api-access-vw4z6\") pod \"scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.348652 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.348580    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:44.484260 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.484232    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"]
May 26 16:15:44.486756 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:15:44.486730    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638600b9_bf0b_4b64_abae_46fe978d90d8.slice/crio-300e0dd6fee0250b548ff2bf51911439899b25fd36418b375309e665912eaf66 WatchSource:0}: Error finding container 300e0dd6fee0250b548ff2bf51911439899b25fd36418b375309e665912eaf66: Status 404 returned error can't find the container with id 300e0dd6fee0250b548ff2bf51911439899b25fd36418b375309e665912eaf66
May 26 16:15:44.537870 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:44.537838    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" event={"ID":"638600b9-bf0b-4b64-abae-46fe978d90d8","Type":"ContainerStarted","Data":"300e0dd6fee0250b548ff2bf51911439899b25fd36418b375309e665912eaf66"}
May 26 16:15:45.543491 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:45.543456    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" event={"ID":"638600b9-bf0b-4b64-abae-46fe978d90d8","Type":"ContainerStarted","Data":"7bb7d844d5c9bed6f7cb22346c815a1f705d49e0b623c7aa9f5e471708af5928"}
May 26 16:15:49.563129 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:49.563089    2561 generic.go:358] "Generic (PLEG): container finished" podID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerID="7bb7d844d5c9bed6f7cb22346c815a1f705d49e0b623c7aa9f5e471708af5928" exitCode=0
May 26 16:15:49.563607 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:49.563167    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" event={"ID":"638600b9-bf0b-4b64-abae-46fe978d90d8","Type":"ContainerDied","Data":"7bb7d844d5c9bed6f7cb22346c815a1f705d49e0b623c7aa9f5e471708af5928"}
May 26 16:15:50.568481 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:50.568446    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" event={"ID":"638600b9-bf0b-4b64-abae-46fe978d90d8","Type":"ContainerStarted","Data":"e26e4a0975ff1095eebf0fccd11c907617187dbbacaab72246b0cd0158cc666c"}
May 26 16:15:50.587228 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:50.587179    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" podStartSLOduration=7.587166359 podStartE2EDuration="7.587166359s" podCreationTimestamp="2026-05-26 16:15:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:15:50.58579988 +0000 UTC m=+897.458058314" watchObservedRunningTime="2026-05-26 16:15:50.587166359 +0000 UTC m=+897.459424792"
May 26 16:15:53.642197 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:53.642168    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:15:53.642702 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:53.642614    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:15:54.349555 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:54.349516    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:54.349739 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:54.349568    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:54.362136 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:54.362113    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:15:54.591841 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:15:54.591816    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:16:34.513703 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.513672    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"]
May 26 16:16:34.514208 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.513980    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" podUID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerName="main" containerID="cri-o://e26e4a0975ff1095eebf0fccd11c907617187dbbacaab72246b0cd0158cc666c" gracePeriod=30
May 26 16:16:34.581136 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.581081    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" podUID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": dial tcp 10.134.0.31:8000: connect: connection refused"
May 26 16:16:34.718992 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.718944    2561 generic.go:358] "Generic (PLEG): container finished" podID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerID="e26e4a0975ff1095eebf0fccd11c907617187dbbacaab72246b0cd0158cc666c" exitCode=0
May 26 16:16:34.718992 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.718993    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" event={"ID":"638600b9-bf0b-4b64-abae-46fe978d90d8","Type":"ContainerDied","Data":"e26e4a0975ff1095eebf0fccd11c907617187dbbacaab72246b0cd0158cc666c"}
May 26 16:16:34.762420 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.762391    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:16:34.840904 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.840867    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638600b9-bf0b-4b64-abae-46fe978d90d8-tls-certs\") pod \"638600b9-bf0b-4b64-abae-46fe978d90d8\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") "
May 26 16:16:34.841103 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.840924    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-dshm\") pod \"638600b9-bf0b-4b64-abae-46fe978d90d8\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") "
May 26 16:16:34.841103 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.840949    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-tmp-dir\") pod \"638600b9-bf0b-4b64-abae-46fe978d90d8\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") "
May 26 16:16:34.841103 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.840971    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-home\") pod \"638600b9-bf0b-4b64-abae-46fe978d90d8\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") "
May 26 16:16:34.841103 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.840988    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-model-cache\") pod \"638600b9-bf0b-4b64-abae-46fe978d90d8\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") "
May 26 16:16:34.841103 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.841029    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/638600b9-bf0b-4b64-abae-46fe978d90d8-kube-api-access-vw4z6\") pod \"638600b9-bf0b-4b64-abae-46fe978d90d8\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") "
May 26 16:16:34.841103 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.841060    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-kserve-provision-location\") pod \"638600b9-bf0b-4b64-abae-46fe978d90d8\" (UID: \"638600b9-bf0b-4b64-abae-46fe978d90d8\") "
May 26 16:16:34.841427 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.841330    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-home" (OuterVolumeSpecName: "home") pod "638600b9-bf0b-4b64-abae-46fe978d90d8" (UID: "638600b9-bf0b-4b64-abae-46fe978d90d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:34.841427 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.841358    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "638600b9-bf0b-4b64-abae-46fe978d90d8" (UID: "638600b9-bf0b-4b64-abae-46fe978d90d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:34.841612 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.841584    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "638600b9-bf0b-4b64-abae-46fe978d90d8" (UID: "638600b9-bf0b-4b64-abae-46fe978d90d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:34.841684 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.841590    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:16:34.841684 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.841639    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:16:34.843317 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.843281    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638600b9-bf0b-4b64-abae-46fe978d90d8-kube-api-access-vw4z6" (OuterVolumeSpecName: "kube-api-access-vw4z6") pod "638600b9-bf0b-4b64-abae-46fe978d90d8" (UID: "638600b9-bf0b-4b64-abae-46fe978d90d8"). InnerVolumeSpecName "kube-api-access-vw4z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:16:34.843609 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.843585    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638600b9-bf0b-4b64-abae-46fe978d90d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "638600b9-bf0b-4b64-abae-46fe978d90d8" (UID: "638600b9-bf0b-4b64-abae-46fe978d90d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:16:34.843609 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.843583    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-dshm" (OuterVolumeSpecName: "dshm") pod "638600b9-bf0b-4b64-abae-46fe978d90d8" (UID: "638600b9-bf0b-4b64-abae-46fe978d90d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:34.896279 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.896221    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "638600b9-bf0b-4b64-abae-46fe978d90d8" (UID: "638600b9-bf0b-4b64-abae-46fe978d90d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:16:34.942969 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.942938    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/638600b9-bf0b-4b64-abae-46fe978d90d8-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:16:34.942969 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.942962    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:16:34.942969 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.942971    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:16:34.943173 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.942980    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw4z6\" (UniqueName: \"kubernetes.io/projected/638600b9-bf0b-4b64-abae-46fe978d90d8-kube-api-access-vw4z6\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:16:34.943173 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:34.942991    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/638600b9-bf0b-4b64-abae-46fe978d90d8-kserve-provision-location\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:16:35.723840 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:35.723814    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"
May 26 16:16:35.723840 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:35.723826    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm" event={"ID":"638600b9-bf0b-4b64-abae-46fe978d90d8","Type":"ContainerDied","Data":"300e0dd6fee0250b548ff2bf51911439899b25fd36418b375309e665912eaf66"}
May 26 16:16:35.724311 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:35.723879    2561 scope.go:117] "RemoveContainer" containerID="e26e4a0975ff1095eebf0fccd11c907617187dbbacaab72246b0cd0158cc666c"
May 26 16:16:35.731833 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:35.731816    2561 scope.go:117] "RemoveContainer" containerID="7bb7d844d5c9bed6f7cb22346c815a1f705d49e0b623c7aa9f5e471708af5928"
May 26 16:16:35.740614 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:35.740591    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"]
May 26 16:16:35.745792 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:35.745766    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-7867b96c5c-gzjrm"]
May 26 16:16:37.711618 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:37.711585    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="638600b9-bf0b-4b64-abae-46fe978d90d8" path="/var/lib/kubelet/pods/638600b9-bf0b-4b64-abae-46fe978d90d8/volumes"
May 26 16:16:41.274078 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.274044    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"]
May 26 16:16:41.274565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.274502    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerName="storage-initializer"
May 26 16:16:41.274565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.274521    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerName="storage-initializer"
May 26 16:16:41.274565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.274532    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerName="main"
May 26 16:16:41.274565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.274540    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerName="main"
May 26 16:16:41.274783 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.274648    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="638600b9-bf0b-4b64-abae-46fe978d90d8" containerName="main"
May 26 16:16:41.276822 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.276794    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.279579 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.279549    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:16:41.280430 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.280411    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:16:41.280654 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.280636    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
May 26 16:16:41.280746 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.280669    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:16:41.288654 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.288629    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"]
May 26 16:16:41.400434 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.400400    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-dshm\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.400434 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.400439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjrpx\" (UniqueName: \"kubernetes.io/projected/2338525a-5955-4d6d-9189-f851fa3e024b-kube-api-access-kjrpx\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.400701 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.400465    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-home\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.400701 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.400481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.400701 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.400591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.400701 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.400658    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2338525a-5955-4d6d-9189-f851fa3e024b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.400701 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.400689    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.494065 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.494031    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"]
May 26 16:16:41.496489 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.496472    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.499970 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.499946    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-54c75\""
May 26 16:16:41.501318 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501299    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2338525a-5955-4d6d-9189-f851fa3e024b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.501414 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501334    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.501496 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501410    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-dshm\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.501496 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501442    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kjrpx\" (UniqueName: \"kubernetes.io/projected/2338525a-5955-4d6d-9189-f851fa3e024b-kube-api-access-kjrpx\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.501496 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501490    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-home\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.501639 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501517    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.501639 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501581    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.501988 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501960    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-home\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.502077 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.501987    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.502077 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.502013    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.502233 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.502210    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.503778 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.503756    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-dshm\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.504033 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.503998    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2338525a-5955-4d6d-9189-f851fa3e024b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.508052 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.508029    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"]
May 26 16:16:41.522183 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.522160    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjrpx\" (UniqueName: \"kubernetes.io/projected/2338525a-5955-4d6d-9189-f851fa3e024b-kube-api-access-kjrpx\") pod \"scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.588506 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.588419    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:16:41.602502 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.602472    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdd5t\" (UniqueName: \"kubernetes.io/projected/88eae174-9483-4b86-a0a8-7e3116bd38a1-kube-api-access-cdd5t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.602607 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.602525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88eae174-9483-4b86-a0a8-7e3116bd38a1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.602671 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.602603    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.602729 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.602665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.602820 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.602797    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.602975 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.602861    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.704488 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.704448    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.704674 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.704507    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.704674 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.704555    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cdd5t\" (UniqueName: \"kubernetes.io/projected/88eae174-9483-4b86-a0a8-7e3116bd38a1-kube-api-access-cdd5t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.704674 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.704590    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88eae174-9483-4b86-a0a8-7e3116bd38a1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.704674 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.704611    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.704674 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.704638    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.705015 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.704994    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-cache\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.705077 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.705024    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-tmp\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.705077 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.705059    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-kserve-provision-location\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.705167 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.705088    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.707118 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.707099    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88eae174-9483-4b86-a0a8-7e3116bd38a1-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.712373 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.712352    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdd5t\" (UniqueName: \"kubernetes.io/projected/88eae174-9483-4b86-a0a8-7e3116bd38a1-kube-api-access-cdd5t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.721493 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.721473    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"]
May 26 16:16:41.723469 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:16:41.723448    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2338525a_5955_4d6d_9189_f851fa3e024b.slice/crio-e8a95e94c1abee8c3c21008ff6ef38a63150e3a5c7152efc6e80ca99d244a43a WatchSource:0}: Error finding container e8a95e94c1abee8c3c21008ff6ef38a63150e3a5c7152efc6e80ca99d244a43a: Status 404 returned error can't find the container with id e8a95e94c1abee8c3c21008ff6ef38a63150e3a5c7152efc6e80ca99d244a43a
May 26 16:16:41.746189 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.746163    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j" event={"ID":"2338525a-5955-4d6d-9189-f851fa3e024b","Type":"ContainerStarted","Data":"e8a95e94c1abee8c3c21008ff6ef38a63150e3a5c7152efc6e80ca99d244a43a"}
May 26 16:16:41.816449 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.816406    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:16:41.990440 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:41.990407    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"]
May 26 16:16:41.992333 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:16:41.992306    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88eae174_9483_4b86_a0a8_7e3116bd38a1.slice/crio-18ca0e05810c745c2efb06d53722482f3a7959dc6fc62d07df0fcbfa9b8a460f WatchSource:0}: Error finding container 18ca0e05810c745c2efb06d53722482f3a7959dc6fc62d07df0fcbfa9b8a460f: Status 404 returned error can't find the container with id 18ca0e05810c745c2efb06d53722482f3a7959dc6fc62d07df0fcbfa9b8a460f
May 26 16:16:42.751944 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:42.751915    2561 generic.go:358] "Generic (PLEG): container finished" podID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerID="3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1" exitCode=0
May 26 16:16:42.752383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:42.752065    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" event={"ID":"88eae174-9483-4b86-a0a8-7e3116bd38a1","Type":"ContainerDied","Data":"3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1"}
May 26 16:16:42.752383 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:42.752101    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" event={"ID":"88eae174-9483-4b86-a0a8-7e3116bd38a1","Type":"ContainerStarted","Data":"18ca0e05810c745c2efb06d53722482f3a7959dc6fc62d07df0fcbfa9b8a460f"}
May 26 16:16:42.754639 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:42.754601    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j" event={"ID":"2338525a-5955-4d6d-9189-f851fa3e024b","Type":"ContainerStarted","Data":"1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5"}
May 26 16:16:44.765309 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:16:44.765265    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" event={"ID":"88eae174-9483-4b86-a0a8-7e3116bd38a1","Type":"ContainerStarted","Data":"5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9"}
May 26 16:17:13.876372 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:13.876333    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" event={"ID":"88eae174-9483-4b86-a0a8-7e3116bd38a1","Type":"ContainerStarted","Data":"32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b"}
May 26 16:17:13.877005 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:13.876525    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:17:13.879428 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:13.879401    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:17:13.896678 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:13.896614    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podStartSLOduration=2.579464347 podStartE2EDuration="32.896594391s" podCreationTimestamp="2026-05-26 16:16:41 +0000 UTC" firstStartedPulling="2026-05-26 16:16:42.753566309 +0000 UTC m=+949.625824734" lastFinishedPulling="2026-05-26 16:17:13.070696366 +0000 UTC m=+979.942954778" observedRunningTime="2026-05-26 16:17:13.896078559 +0000 UTC m=+980.768336994" watchObservedRunningTime="2026-05-26 16:17:13.896594391 +0000 UTC m=+980.768852842"
May 26 16:17:14.882561 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:14.882524    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:17:21.817431 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:21.817387    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:17:21.818037 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:21.817443    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:17:21.819464 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:21.819420    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:17:21.819928 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:21.819876    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:17:21.904750 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:21.904718    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:17:21.904974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:21.904954    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:17:22.909179 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:22.909138    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:17:32.909539 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:32.909503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:17:42.909873 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:42.909833    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:17:52.909493 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:17:52.909458    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:18:02.909733 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:18:02.909691    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:18:12.909485 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:18:12.909447    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:18:22.909978 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:18:22.909938    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:18:32.909786 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:18:32.909749    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:18:42.909842 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:18:42.909753    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:18:49.709153 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:18:49.709119    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:18:59.709302 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:18:59.709256    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:19:09.709425 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:19:09.709386    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:19:19.709553 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:19:19.709518    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:19:29.710091 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:19:29.710056    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:19:39.709556 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:19:39.709503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:19:49.709208 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:19:49.709169    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:19:59.709118 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:19:59.709083    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:20:03.712236 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:20:03.712175    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:20:13.711945 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:20:13.711832    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:20:23.712044 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:20:23.712009    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:20:33.712011 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:20:33.711975    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:20:43.712081 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:20:43.712042    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:20:53.669719 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:20:53.669687    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:20:53.670248 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:20:53.670229    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:20:53.712362 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:20:53.712327    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:21:03.712172 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:21:03.712136    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:21:13.712455 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:21:13.712412    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:21:23.712394 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:21:23.712357    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:21:27.709579 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:21:27.709539    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:21:37.709973 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:21:37.709922    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:21:47.710347 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:21:47.710257    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:21:57.710104 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:21:57.710063    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:22:07.710084 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:22:07.710048    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:22:17.709816 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:22:17.709775    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:22:27.709557 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:22:27.709519    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:22:37.709974 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:22:37.709928    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:22:47.710055 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:22:47.710021    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:22:57.709466 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:22:57.709421    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:23:07.709269 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:23:07.709226    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:23:17.709425 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:23:17.709324    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:23:27.709856 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:23:27.709817    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:23:37.709469 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:23:37.709429    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:23:47.709857 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:23:47.709821    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:23:57.709253 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:23:57.709219    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:24:04.709132 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:24:04.709096    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:24:14.710139 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:24:14.710091    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:24:24.710006 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:24:24.709967    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:24:34.709251 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:24:34.709205    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:24:44.710200 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:24:44.710118    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:24:54.710045 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:24:54.709996    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:25:04.709114 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:04.709079    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:25:14.709196 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:14.709161    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:25:24.710083 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:24.710045    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:25:28.708935 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:28.708884    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:25:38.708946 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:38.708905    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:25:48.709620 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:48.709578    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:25:53.693514 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:53.693486    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:25:53.695608 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:53.695588    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:25:58.709591 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:25:58.709544    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:26:08.708995 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:26:08.708955    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:26:18.709312 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:26:18.709273    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:26:28.709084 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:26:28.709040    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:26:38.709091 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:26:38.709049    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:26:40.709402 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:26:40.709364    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:26:50.709520 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:26:50.709479    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:27:00.710252 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:27:00.710214    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:27:10.709611 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:27:10.709574    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:27:20.709949 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:27:20.709910    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:27:30.709966 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:27:30.709929    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:27:40.709955 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:27:40.709862    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:27:50.709350 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:27:50.709310    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:27:51.708818 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:27:51.708777    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:28:01.709033 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:28:01.708994    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:28:11.708986 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:28:11.708945    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:28:21.709414 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:28:21.709372    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:28:31.709332 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:28:31.709295    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:28:41.709353 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:28:41.709310    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:28:51.709107 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:28:51.709064    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:29:01.708930 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:29:01.708862    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:29:02.708875 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:29:02.708834    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:29:12.708871 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:29:12.708788    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:29:22.709550 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:29:22.709511    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:29:32.709762 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:29:32.709726    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:29:42.709171 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:29:42.709135    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:29:52.709545 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:29:52.709503    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:30:02.708761 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:02.708721    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:30:12.709537 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:12.709496    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:30:22.709241 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:22.709206    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:30:24.709633 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:24.709595    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:30:34.709629 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:34.709583    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:30:44.710400 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:44.710310    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:30:53.717186 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:53.717158    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:30:53.720106 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:53.720087    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:30:54.710512 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:30:54.710474    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:31:04.709442 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:04.709400    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:31:14.710292 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:14.710253    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:31:24.710337 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:24.710295    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:31:34.709561 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:34.709516    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:31:35.131184 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:35.131146    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"]
May 26 16:31:35.131695 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:35.131641    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" containerID="cri-o://5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9" gracePeriod=30
May 26 16:31:35.131802 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:35.131733    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="tokenizer" containerID="cri-o://32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b" gracePeriod=30
May 26 16:31:35.136562 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:35.136518    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
May 26 16:31:35.139739 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:35.139681    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"]
May 26 16:31:35.140260 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:35.140206    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j" podUID="2338525a-5955-4d6d-9189-f851fa3e024b" containerName="storage-initializer" containerID="cri-o://1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5" gracePeriod=30
May 26 16:31:35.775513 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:35.775478    2561 generic.go:358] "Generic (PLEG): container finished" podID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerID="5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9" exitCode=0
May 26 16:31:35.775881 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:35.775525    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" event={"ID":"88eae174-9483-4b86-a0a8-7e3116bd38a1","Type":"ContainerDied","Data":"5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9"}
May 26 16:31:36.490669 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.490644    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:31:36.567507 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.567470    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdd5t\" (UniqueName: \"kubernetes.io/projected/88eae174-9483-4b86-a0a8-7e3116bd38a1-kube-api-access-cdd5t\") pod \"88eae174-9483-4b86-a0a8-7e3116bd38a1\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") "
May 26 16:31:36.567507 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.567511    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-kserve-provision-location\") pod \"88eae174-9483-4b86-a0a8-7e3116bd38a1\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") "
May 26 16:31:36.567737 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.567551    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-cache\") pod \"88eae174-9483-4b86-a0a8-7e3116bd38a1\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") "
May 26 16:31:36.567737 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.567599    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88eae174-9483-4b86-a0a8-7e3116bd38a1-tls-certs\") pod \"88eae174-9483-4b86-a0a8-7e3116bd38a1\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") "
May 26 16:31:36.567737 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.567659    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-tmp\") pod \"88eae174-9483-4b86-a0a8-7e3116bd38a1\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") "
May 26 16:31:36.567737 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.567684    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-uds\") pod \"88eae174-9483-4b86-a0a8-7e3116bd38a1\" (UID: \"88eae174-9483-4b86-a0a8-7e3116bd38a1\") "
May 26 16:31:36.567982 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.567868    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "88eae174-9483-4b86-a0a8-7e3116bd38a1" (UID: "88eae174-9483-4b86-a0a8-7e3116bd38a1"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:36.568065 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.568041    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "88eae174-9483-4b86-a0a8-7e3116bd38a1" (UID: "88eae174-9483-4b86-a0a8-7e3116bd38a1"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:36.568140 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.568059    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "88eae174-9483-4b86-a0a8-7e3116bd38a1" (UID: "88eae174-9483-4b86-a0a8-7e3116bd38a1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:36.568366 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.568347    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88eae174-9483-4b86-a0a8-7e3116bd38a1" (UID: "88eae174-9483-4b86-a0a8-7e3116bd38a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:31:36.569697 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.569672    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88eae174-9483-4b86-a0a8-7e3116bd38a1-kube-api-access-cdd5t" (OuterVolumeSpecName: "kube-api-access-cdd5t") pod "88eae174-9483-4b86-a0a8-7e3116bd38a1" (UID: "88eae174-9483-4b86-a0a8-7e3116bd38a1"). InnerVolumeSpecName "kube-api-access-cdd5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:31:36.569847 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.569829    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88eae174-9483-4b86-a0a8-7e3116bd38a1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88eae174-9483-4b86-a0a8-7e3116bd38a1" (UID: "88eae174-9483-4b86-a0a8-7e3116bd38a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:31:36.668661 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.668621    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88eae174-9483-4b86-a0a8-7e3116bd38a1-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:31:36.668661 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.668655    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-tmp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:31:36.668661 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.668669    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-uds\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:31:36.668936 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.668683    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdd5t\" (UniqueName: \"kubernetes.io/projected/88eae174-9483-4b86-a0a8-7e3116bd38a1-kube-api-access-cdd5t\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:31:36.668936 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.668696    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-kserve-provision-location\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:31:36.668936 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.668708    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/88eae174-9483-4b86-a0a8-7e3116bd38a1-tokenizer-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:31:36.780306 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.780268    2561 generic.go:358] "Generic (PLEG): container finished" podID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerID="32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b" exitCode=0
May 26 16:31:36.780753 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.780329    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" event={"ID":"88eae174-9483-4b86-a0a8-7e3116bd38a1","Type":"ContainerDied","Data":"32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b"}
May 26 16:31:36.780753 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.780339    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"
May 26 16:31:36.780753 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.780362    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj" event={"ID":"88eae174-9483-4b86-a0a8-7e3116bd38a1","Type":"ContainerDied","Data":"18ca0e05810c745c2efb06d53722482f3a7959dc6fc62d07df0fcbfa9b8a460f"}
May 26 16:31:36.780753 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.780378    2561 scope.go:117] "RemoveContainer" containerID="32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b"
May 26 16:31:36.789556 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.789537    2561 scope.go:117] "RemoveContainer" containerID="5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9"
May 26 16:31:36.797147 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.797124    2561 scope.go:117] "RemoveContainer" containerID="3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1"
May 26 16:31:36.804136 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.804105    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"]
May 26 16:31:36.804727 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.804705    2561 scope.go:117] "RemoveContainer" containerID="32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b"
May 26 16:31:36.805153 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:31:36.805132    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b\": container with ID starting with 32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b not found: ID does not exist" containerID="32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b"
May 26 16:31:36.805253 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.805164    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b"} err="failed to get container status \"32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b\": rpc error: code = NotFound desc = could not find container \"32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b\": container with ID starting with 32ea74feffa1c06e46ff27017f07f6f4e54814da94cfe23ecd359cb6f7fd6c9b not found: ID does not exist"
May 26 16:31:36.805253 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.805182    2561 scope.go:117] "RemoveContainer" containerID="5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9"
May 26 16:31:36.805462 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:31:36.805447    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9\": container with ID starting with 5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9 not found: ID does not exist" containerID="5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9"
May 26 16:31:36.805505 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.805467    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9"} err="failed to get container status \"5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9\": rpc error: code = NotFound desc = could not find container \"5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9\": container with ID starting with 5c063a28e84014f3d79e734dce17f4d5dc60d83f8c31f5966f483d08cbb25fe9 not found: ID does not exist"
May 26 16:31:36.805505 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.805483    2561 scope.go:117] "RemoveContainer" containerID="3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1"
May 26 16:31:36.805754 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:31:36.805735    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1\": container with ID starting with 3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1 not found: ID does not exist" containerID="3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1"
May 26 16:31:36.805800 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.805760    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1"} err="failed to get container status \"3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1\": rpc error: code = NotFound desc = could not find container \"3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1\": container with ID starting with 3ac55fcfcb2084b861ecc652923eececaeb40bc95bd63fa6ebaf4d13c70f0ba1 not found: ID does not exist"
May 26 16:31:36.808549 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:36.808524    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-5ff945dgvmgj"]
May 26 16:31:37.712773 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:37.712740    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" path="/var/lib/kubelet/pods/88eae174-9483-4b86-a0a8-7e3116bd38a1/volumes"
May 26 16:31:45.799481 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799445    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"]
May 26 16:31:45.799874 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799803    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main"
May 26 16:31:45.799874 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799819    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main"
May 26 16:31:45.799874 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799845    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="tokenizer"
May 26 16:31:45.799874 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799851    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="tokenizer"
May 26 16:31:45.799874 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799858    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="storage-initializer"
May 26 16:31:45.799874 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799865    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="storage-initializer"
May 26 16:31:45.800106 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799937    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="main"
May 26 16:31:45.800106 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.799947    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="88eae174-9483-4b86-a0a8-7e3116bd38a1" containerName="tokenizer"
May 26 16:31:45.804338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.804314    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:45.807156 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.807133    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
May 26 16:31:45.813294 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.813269    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"]
May 26 16:31:45.945863 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.945824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tls-certs\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:45.946076 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.945875    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-dshm\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:45.946076 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.945963    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-home\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:45.946076 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.945988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:45.946076 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.946031    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-model-cache\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:45.946076 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.946071    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sv6l\" (UniqueName: \"kubernetes.io/projected/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kube-api-access-4sv6l\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:45.946301 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:45.946152    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047080 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047043    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-dshm\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047262 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047095    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-home\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047262 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047114    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047262 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047150    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-model-cache\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047262 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047185    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4sv6l\" (UniqueName: \"kubernetes.io/projected/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kube-api-access-4sv6l\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047461 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047283    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047461 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047336    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tls-certs\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047564 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047486    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-home\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047607 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047571    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tmp-dir\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047642 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047612    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-model-cache\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.047722 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.047696    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.049366 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.049348    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-dshm\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.049791 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.049753    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tls-certs\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.056119 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.056097    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sv6l\" (UniqueName: \"kubernetes.io/projected/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kube-api-access-4sv6l\") pod \"precise-prefix-cache-test-kserve-7d6cdc99-5f8v2\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.116699 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.116660    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:46.136371 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.136337    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"]
May 26 16:31:46.141531 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.141503    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.144130 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.144109    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-epp-sa-dockercfg-vlp2n\""
May 26 16:31:46.154592 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.154559    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"]
May 26 16:31:46.249777 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.249745    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9g6\" (UniqueName: \"kubernetes.io/projected/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kube-api-access-6v9g6\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.249958 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.249787    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.249958 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.249853    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.249958 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.249885    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.250066 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.250010    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.250066 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.250050    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.263507 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.263481    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"]
May 26 16:31:46.266332 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:31:46.266305    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9903aa_c2c0_4f7b_aa46_6858e9838019.slice/crio-4fc78f5f9dacbac943973ff413ca2c925ed81bdf9c4036d41e415b0d7b2df679 WatchSource:0}: Error finding container 4fc78f5f9dacbac943973ff413ca2c925ed81bdf9c4036d41e415b0d7b2df679: Status 404 returned error can't find the container with id 4fc78f5f9dacbac943973ff413ca2c925ed81bdf9c4036d41e415b0d7b2df679
May 26 16:31:46.268341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.268322    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:31:46.351146 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351112    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9g6\" (UniqueName: \"kubernetes.io/projected/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kube-api-access-6v9g6\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351161    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351186    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351204    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351265    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351338 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351323    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351633 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351602    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351693 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351654    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-cache\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351758 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351695    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-tmp\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.351849 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.351829    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-uds\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.353779 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.353753    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tls-certs\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.362086 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.362060    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9g6\" (UniqueName: \"kubernetes.io/projected/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kube-api-access-6v9g6\") pod \"precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.455328 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.455290    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:46.613165 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.613135    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"]
May 26 16:31:46.616281 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:31:46.616232    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ffecdb_9d5e_4f6e_ad7a_7174abee325c.slice/crio-2208564eacc9cb7f8c9e4a95b631f712925c6a56c676e7900fe59ddbc374b594 WatchSource:0}: Error finding container 2208564eacc9cb7f8c9e4a95b631f712925c6a56c676e7900fe59ddbc374b594: Status 404 returned error can't find the container with id 2208564eacc9cb7f8c9e4a95b631f712925c6a56c676e7900fe59ddbc374b594
May 26 16:31:46.814201 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.814160    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" event={"ID":"22ffecdb-9d5e-4f6e-ad7a-7174abee325c","Type":"ContainerStarted","Data":"4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe"}
May 26 16:31:46.814201 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.814205    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" event={"ID":"22ffecdb-9d5e-4f6e-ad7a-7174abee325c","Type":"ContainerStarted","Data":"2208564eacc9cb7f8c9e4a95b631f712925c6a56c676e7900fe59ddbc374b594"}
May 26 16:31:46.815595 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.815563    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2" event={"ID":"7e9903aa-c2c0-4f7b-aa46-6858e9838019","Type":"ContainerStarted","Data":"02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e"}
May 26 16:31:46.815595 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:46.815598    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2" event={"ID":"7e9903aa-c2c0-4f7b-aa46-6858e9838019","Type":"ContainerStarted","Data":"4fc78f5f9dacbac943973ff413ca2c925ed81bdf9c4036d41e415b0d7b2df679"}
May 26 16:31:47.820592 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:47.820553    2561 generic.go:358] "Generic (PLEG): container finished" podID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerID="4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe" exitCode=0
May 26 16:31:47.821547 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:47.820637    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" event={"ID":"22ffecdb-9d5e-4f6e-ad7a-7174abee325c","Type":"ContainerDied","Data":"4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe"}
May 26 16:31:48.826380 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:48.826341    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" event={"ID":"22ffecdb-9d5e-4f6e-ad7a-7174abee325c","Type":"ContainerStarted","Data":"83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6"}
May 26 16:31:48.826380 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:48.826386    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" event={"ID":"22ffecdb-9d5e-4f6e-ad7a-7174abee325c","Type":"ContainerStarted","Data":"62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837"}
May 26 16:31:48.826826 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:48.826458    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:48.854513 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:48.854445    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" podStartSLOduration=2.854421776 podStartE2EDuration="2.854421776s" podCreationTimestamp="2026-05-26 16:31:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:31:48.851854184 +0000 UTC m=+1855.724112632" watchObservedRunningTime="2026-05-26 16:31:48.854421776 +0000 UTC m=+1855.726680262"
May 26 16:31:50.835420 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:50.835382    2561 generic.go:358] "Generic (PLEG): container finished" podID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" containerID="02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e" exitCode=0
May 26 16:31:50.837745 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:50.835454    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2" event={"ID":"7e9903aa-c2c0-4f7b-aa46-6858e9838019","Type":"ContainerDied","Data":"02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e"}
May 26 16:31:51.840953 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:51.840912    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2" event={"ID":"7e9903aa-c2c0-4f7b-aa46-6858e9838019","Type":"ContainerStarted","Data":"59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e"}
May 26 16:31:51.863144 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:51.863089    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2" podStartSLOduration=6.863071746 podStartE2EDuration="6.863071746s" podCreationTimestamp="2026-05-26 16:31:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:31:51.860407652 +0000 UTC m=+1858.732666082" watchObservedRunningTime="2026-05-26 16:31:51.863071746 +0000 UTC m=+1858.735330222"
May 26 16:31:56.117476 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:56.117435    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:56.117999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:56.117486    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:56.129854 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:56.129832    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:56.456219 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:56.456126    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:56.456605 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:56.456577    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:56.457551 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:31:56.457521    2561 logging.go:55] [core] [Channel #566 SubChannel #567]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.35:9003", ServerName: "10.134.0.35:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.35:9003: connect: connection refused"
May 26 16:31:56.458877 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:56.458853    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:56.862304 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:56.862270    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:31:56.872660 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:56.872633    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:31:57.457322 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:31:57.457267    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.35:9003\" within 1s: context deadline exceeded"
May 26 16:32:05.378686 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.378659    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j_2338525a-5955-4d6d-9189-f851fa3e024b/storage-initializer/0.log"
May 26 16:32:05.379039 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.378750    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:32:05.526369 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526341    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-model-cache\") pod \"2338525a-5955-4d6d-9189-f851fa3e024b\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") "
May 26 16:32:05.526528 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526402    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2338525a-5955-4d6d-9189-f851fa3e024b-tls-certs\") pod \"2338525a-5955-4d6d-9189-f851fa3e024b\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") "
May 26 16:32:05.526528 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526431    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-kserve-provision-location\") pod \"2338525a-5955-4d6d-9189-f851fa3e024b\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") "
May 26 16:32:05.526528 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526489    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-dshm\") pod \"2338525a-5955-4d6d-9189-f851fa3e024b\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") "
May 26 16:32:05.526682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526530    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjrpx\" (UniqueName: \"kubernetes.io/projected/2338525a-5955-4d6d-9189-f851fa3e024b-kube-api-access-kjrpx\") pod \"2338525a-5955-4d6d-9189-f851fa3e024b\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") "
May 26 16:32:05.526682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526584    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-home\") pod \"2338525a-5955-4d6d-9189-f851fa3e024b\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") "
May 26 16:32:05.526682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526619    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-tmp-dir\") pod \"2338525a-5955-4d6d-9189-f851fa3e024b\" (UID: \"2338525a-5955-4d6d-9189-f851fa3e024b\") "
May 26 16:32:05.526682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526637    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2338525a-5955-4d6d-9189-f851fa3e024b" (UID: "2338525a-5955-4d6d-9189-f851fa3e024b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:32:05.526988 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.526958    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:32:05.527138 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.527114    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-home" (OuterVolumeSpecName: "home") pod "2338525a-5955-4d6d-9189-f851fa3e024b" (UID: "2338525a-5955-4d6d-9189-f851fa3e024b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:32:05.527210 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.527159    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2338525a-5955-4d6d-9189-f851fa3e024b" (UID: "2338525a-5955-4d6d-9189-f851fa3e024b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:32:05.528783 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.528758    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2338525a-5955-4d6d-9189-f851fa3e024b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2338525a-5955-4d6d-9189-f851fa3e024b" (UID: "2338525a-5955-4d6d-9189-f851fa3e024b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:32:05.528922 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.528826    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2338525a-5955-4d6d-9189-f851fa3e024b-kube-api-access-kjrpx" (OuterVolumeSpecName: "kube-api-access-kjrpx") pod "2338525a-5955-4d6d-9189-f851fa3e024b" (UID: "2338525a-5955-4d6d-9189-f851fa3e024b"). InnerVolumeSpecName "kube-api-access-kjrpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:32:05.529038 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.529021    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-dshm" (OuterVolumeSpecName: "dshm") pod "2338525a-5955-4d6d-9189-f851fa3e024b" (UID: "2338525a-5955-4d6d-9189-f851fa3e024b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:32:05.565000 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.564961    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2338525a-5955-4d6d-9189-f851fa3e024b" (UID: "2338525a-5955-4d6d-9189-f851fa3e024b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:32:05.628441 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.628402    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:32:05.628441 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.628436    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjrpx\" (UniqueName: \"kubernetes.io/projected/2338525a-5955-4d6d-9189-f851fa3e024b-kube-api-access-kjrpx\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:32:05.628441 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.628449    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:32:05.628672 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.628460    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:32:05.628672 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.628469    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2338525a-5955-4d6d-9189-f851fa3e024b-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:32:05.628672 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.628477    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2338525a-5955-4d6d-9189-f851fa3e024b-kserve-provision-location\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:32:05.897471 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.897385    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j_2338525a-5955-4d6d-9189-f851fa3e024b/storage-initializer/0.log"
May 26 16:32:05.897471 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.897429    2561 generic.go:358] "Generic (PLEG): container finished" podID="2338525a-5955-4d6d-9189-f851fa3e024b" containerID="1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5" exitCode=137
May 26 16:32:05.897661 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.897477    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j" event={"ID":"2338525a-5955-4d6d-9189-f851fa3e024b","Type":"ContainerDied","Data":"1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5"}
May 26 16:32:05.897661 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.897506    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j" event={"ID":"2338525a-5955-4d6d-9189-f851fa3e024b","Type":"ContainerDied","Data":"e8a95e94c1abee8c3c21008ff6ef38a63150e3a5c7152efc6e80ca99d244a43a"}
May 26 16:32:05.897661 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.897511    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"
May 26 16:32:05.897661 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.897525    2561 scope.go:117] "RemoveContainer" containerID="1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5"
May 26 16:32:05.931461 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.931417    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"]
May 26 16:32:05.934020 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.933991    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-58fb968c6f-thg2j"]
May 26 16:32:05.948228 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.948202    2561 scope.go:117] "RemoveContainer" containerID="1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5"
May 26 16:32:05.948614 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:32:05.948579    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5\": container with ID starting with 1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5 not found: ID does not exist" containerID="1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5"
May 26 16:32:05.948757 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:05.948624    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5"} err="failed to get container status \"1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5\": rpc error: code = NotFound desc = could not find container \"1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5\": container with ID starting with 1bb185d3deedf189cf43b676db884169fb6f855d7adbea74bff044d11a24e7d5 not found: ID does not exist"
May 26 16:32:06.456708 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:32:06.456677    2561 logging.go:55] [core] [Channel #568 SubChannel #569]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.35:9003", ServerName: "10.134.0.35:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.35:9003: connect: connection refused"
May 26 16:32:07.457152 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:07.457114    2561 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.35:9003\" within 1s: context deadline exceeded"
May 26 16:32:07.457615 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:32:07.457233    2561 logging.go:55] [core] [Channel #568 SubChannel #569]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.35:9003", ServerName: "10.134.0.35:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.35:9003: connect: connection refused"
May 26 16:32:07.712673 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:07.712576    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2338525a-5955-4d6d-9189-f851fa3e024b" path="/var/lib/kubelet/pods/2338525a-5955-4d6d-9189-f851fa3e024b/volumes"
May 26 16:32:18.870843 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:32:18.870769    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:33:38.298611 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.298575    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"]
May 26 16:33:38.299175 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.298851    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2" podUID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" containerName="main" containerID="cri-o://59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e" gracePeriod=30
May 26 16:33:38.305859 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.305739    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"]
May 26 16:33:38.306238 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.306181    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="main" containerID="cri-o://62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837" gracePeriod=30
May 26 16:33:38.306336 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.306283    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="tokenizer" containerID="cri-o://83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6" gracePeriod=30
May 26 16:33:38.561451 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.561384    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:33:38.657279 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657245    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kserve-provision-location\") pod \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") "
May 26 16:33:38.657474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657292    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tmp-dir\") pod \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") "
May 26 16:33:38.657474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657310    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-model-cache\") pod \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") "
May 26 16:33:38.657474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657348    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-dshm\") pod \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") "
May 26 16:33:38.657474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657373    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sv6l\" (UniqueName: \"kubernetes.io/projected/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kube-api-access-4sv6l\") pod \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") "
May 26 16:33:38.657474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657405    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tls-certs\") pod \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") "
May 26 16:33:38.657474 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657441    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-home\") pod \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\" (UID: \"7e9903aa-c2c0-4f7b-aa46-6858e9838019\") "
May 26 16:33:38.657780 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657561    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e9903aa-c2c0-4f7b-aa46-6858e9838019" (UID: "7e9903aa-c2c0-4f7b-aa46-6858e9838019"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:38.657780 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657587    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e9903aa-c2c0-4f7b-aa46-6858e9838019" (UID: "7e9903aa-c2c0-4f7b-aa46-6858e9838019"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:38.657878 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.657849    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-home" (OuterVolumeSpecName: "home") pod "7e9903aa-c2c0-4f7b-aa46-6858e9838019" (UID: "7e9903aa-c2c0-4f7b-aa46-6858e9838019"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:38.658151 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.658119    2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tmp-dir\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:38.658151 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.658142    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-model-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:38.658151 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.658153    2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-home\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:38.659788 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.659754    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e9903aa-c2c0-4f7b-aa46-6858e9838019" (UID: "7e9903aa-c2c0-4f7b-aa46-6858e9838019"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:33:38.660060 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.660028    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kube-api-access-4sv6l" (OuterVolumeSpecName: "kube-api-access-4sv6l") pod "7e9903aa-c2c0-4f7b-aa46-6858e9838019" (UID: "7e9903aa-c2c0-4f7b-aa46-6858e9838019"). InnerVolumeSpecName "kube-api-access-4sv6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:33:38.660150 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.660122    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-dshm" (OuterVolumeSpecName: "dshm") pod "7e9903aa-c2c0-4f7b-aa46-6858e9838019" (UID: "7e9903aa-c2c0-4f7b-aa46-6858e9838019"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:38.714581 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.714526    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7e9903aa-c2c0-4f7b-aa46-6858e9838019" (UID: "7e9903aa-c2c0-4f7b-aa46-6858e9838019"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:38.758877 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.758839    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kserve-provision-location\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:38.758877 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.758869    2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e9903aa-c2c0-4f7b-aa46-6858e9838019-dshm\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:38.758877 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.758881    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sv6l\" (UniqueName: \"kubernetes.io/projected/7e9903aa-c2c0-4f7b-aa46-6858e9838019-kube-api-access-4sv6l\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:38.758877 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:38.758905    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e9903aa-c2c0-4f7b-aa46-6858e9838019-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:38.870052 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:33:38.869969    2561 logging.go:55] [core] [Channel #621 SubChannel #622]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.35:9003", ServerName: "10.134.0.35:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.35:9003: connect: connection refused"
May 26 16:33:39.213378 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.213258    2561 generic.go:358] "Generic (PLEG): container finished" podID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerID="62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837" exitCode=0
May 26 16:33:39.213378 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.213339    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" event={"ID":"22ffecdb-9d5e-4f6e-ad7a-7174abee325c","Type":"ContainerDied","Data":"62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837"}
May 26 16:33:39.214812 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.214782    2561 generic.go:358] "Generic (PLEG): container finished" podID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" containerID="59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e" exitCode=0
May 26 16:33:39.214977 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.214838    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2" event={"ID":"7e9903aa-c2c0-4f7b-aa46-6858e9838019","Type":"ContainerDied","Data":"59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e"}
May 26 16:33:39.214977 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.214862    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2" event={"ID":"7e9903aa-c2c0-4f7b-aa46-6858e9838019","Type":"ContainerDied","Data":"4fc78f5f9dacbac943973ff413ca2c925ed81bdf9c4036d41e415b0d7b2df679"}
May 26 16:33:39.214977 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.214866    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"
May 26 16:33:39.214977 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.214878    2561 scope.go:117] "RemoveContainer" containerID="59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e"
May 26 16:33:39.223682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.223666    2561 scope.go:117] "RemoveContainer" containerID="02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e"
May 26 16:33:39.238184 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.238149    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"]
May 26 16:33:39.239690 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.239670    2561 scope.go:117] "RemoveContainer" containerID="59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e"
May 26 16:33:39.240376 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:33:39.240199    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e\": container with ID starting with 59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e not found: ID does not exist" containerID="59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e"
May 26 16:33:39.240376 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.240235    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e"} err="failed to get container status \"59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e\": rpc error: code = NotFound desc = could not find container \"59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e\": container with ID starting with 59ba96536dedb5637e940d9347ae6cc35615cc8e8532ce619d4e6eb0c210039e not found: ID does not exist"
May 26 16:33:39.240376 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.240261    2561 scope.go:117] "RemoveContainer" containerID="02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e"
May 26 16:33:39.240633 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:33:39.240593    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e\": container with ID starting with 02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e not found: ID does not exist" containerID="02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e"
May 26 16:33:39.240732 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.240645    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e"} err="failed to get container status \"02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e\": rpc error: code = NotFound desc = could not find container \"02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e\": container with ID starting with 02bafd295a6420b90ce514a1baf3bb157d1413ad90484b12eb5691cb9192e03e not found: ID does not exist"
May 26 16:33:39.241164 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.241146    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-7d6cdc99-5f8v2"]
May 26 16:33:39.712852 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.712815    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" path="/var/lib/kubelet/pods/7e9903aa-c2c0-4f7b-aa46-6858e9838019/volumes"
May 26 16:33:39.865438 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.865414    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:33:39.870122 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.870095    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.35:9003\" within 1s: context deadline exceeded"
May 26 16:33:39.969156 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969067    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-uds\") pod \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") "
May 26 16:33:39.969156 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969144    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-tmp\") pod \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") "
May 26 16:33:39.969357 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969170    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-cache\") pod \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") "
May 26 16:33:39.969357 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969199    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tls-certs\") pod \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") "
May 26 16:33:39.969357 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969261    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kserve-provision-location\") pod \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") "
May 26 16:33:39.969357 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969280    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v9g6\" (UniqueName: \"kubernetes.io/projected/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kube-api-access-6v9g6\") pod \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\" (UID: \"22ffecdb-9d5e-4f6e-ad7a-7174abee325c\") "
May 26 16:33:39.969357 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969337    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "22ffecdb-9d5e-4f6e-ad7a-7174abee325c" (UID: "22ffecdb-9d5e-4f6e-ad7a-7174abee325c"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:39.969619 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969503    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "22ffecdb-9d5e-4f6e-ad7a-7174abee325c" (UID: "22ffecdb-9d5e-4f6e-ad7a-7174abee325c"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:39.969682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969626    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-uds\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:39.969682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969650    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-tmp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:39.969682 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.969533    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "22ffecdb-9d5e-4f6e-ad7a-7174abee325c" (UID: "22ffecdb-9d5e-4f6e-ad7a-7174abee325c"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:39.970113 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.970092    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "22ffecdb-9d5e-4f6e-ad7a-7174abee325c" (UID: "22ffecdb-9d5e-4f6e-ad7a-7174abee325c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:33:39.971516 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.971494    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22ffecdb-9d5e-4f6e-ad7a-7174abee325c" (UID: "22ffecdb-9d5e-4f6e-ad7a-7174abee325c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:33:39.971516 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:39.971481    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kube-api-access-6v9g6" (OuterVolumeSpecName: "kube-api-access-6v9g6") pod "22ffecdb-9d5e-4f6e-ad7a-7174abee325c" (UID: "22ffecdb-9d5e-4f6e-ad7a-7174abee325c"). InnerVolumeSpecName "kube-api-access-6v9g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:33:40.070587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.070554    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:40.070587 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.070588    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kserve-provision-location\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:40.070861 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.070604    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9g6\" (UniqueName: \"kubernetes.io/projected/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-kube-api-access-6v9g6\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:40.070861 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.070619    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/22ffecdb-9d5e-4f6e-ad7a-7174abee325c-tokenizer-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:33:40.226090 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.225999    2561 generic.go:358] "Generic (PLEG): container finished" podID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerID="83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6" exitCode=0
May 26 16:33:40.226090 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.226071    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" event={"ID":"22ffecdb-9d5e-4f6e-ad7a-7174abee325c","Type":"ContainerDied","Data":"83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6"}
May 26 16:33:40.226317 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.226096    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx" event={"ID":"22ffecdb-9d5e-4f6e-ad7a-7174abee325c","Type":"ContainerDied","Data":"2208564eacc9cb7f8c9e4a95b631f712925c6a56c676e7900fe59ddbc374b594"}
May 26 16:33:40.226317 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.226096    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"
May 26 16:33:40.226317 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.226111    2561 scope.go:117] "RemoveContainer" containerID="83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6"
May 26 16:33:40.235040 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.235021    2561 scope.go:117] "RemoveContainer" containerID="62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837"
May 26 16:33:40.242656 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.242636    2561 scope.go:117] "RemoveContainer" containerID="4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe"
May 26 16:33:40.249300 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.249277    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"]
May 26 16:33:40.249915 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.249877    2561 scope.go:117] "RemoveContainer" containerID="83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6"
May 26 16:33:40.250189 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:33:40.250159    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6\": container with ID starting with 83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6 not found: ID does not exist" containerID="83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6"
May 26 16:33:40.250253 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.250190    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6"} err="failed to get container status \"83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6\": rpc error: code = NotFound desc = could not find container \"83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6\": container with ID starting with 83ed8bc0ab907a273c076e8c93745685bb7b4801a34170652ecd12736df9fbb6 not found: ID does not exist"
May 26 16:33:40.250253 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.250213    2561 scope.go:117] "RemoveContainer" containerID="62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837"
May 26 16:33:40.250493 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:33:40.250475    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837\": container with ID starting with 62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837 not found: ID does not exist" containerID="62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837"
May 26 16:33:40.250560 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.250502    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837"} err="failed to get container status \"62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837\": rpc error: code = NotFound desc = could not find container \"62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837\": container with ID starting with 62a6aae8a3a717e7724f90eafab85e3dac4734aa86dc8d0e34baf135decf9837 not found: ID does not exist"
May 26 16:33:40.250560 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.250525    2561 scope.go:117] "RemoveContainer" containerID="4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe"
May 26 16:33:40.250801 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:33:40.250782    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe\": container with ID starting with 4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe not found: ID does not exist" containerID="4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe"
May 26 16:33:40.250875 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.250805    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe"} err="failed to get container status \"4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe\": rpc error: code = NotFound desc = could not find container \"4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe\": container with ID starting with 4f5e86f758df1addb185d9a9635a88e62b049878b982613906702de121b50bbe not found: ID does not exist"
May 26 16:33:40.253706 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:40.253684    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-router-scheduler-75f8c749dvlnx"]
May 26 16:33:41.712303 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:41.712266    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" path="/var/lib/kubelet/pods/22ffecdb-9d5e-4f6e-ad7a-7174abee325c/volumes"
May 26 16:33:58.737541 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737501    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"]
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737906    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="tokenizer"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737925    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="tokenizer"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737939    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="main"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737945    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="main"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737957    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" containerName="main"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737963    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" containerName="main"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737970    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="storage-initializer"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737976    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="storage-initializer"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737987    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" containerName="storage-initializer"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.737995    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" containerName="storage-initializer"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.738009    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2338525a-5955-4d6d-9189-f851fa3e024b" containerName="storage-initializer"
May 26 16:33:58.738043 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.738016    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2338525a-5955-4d6d-9189-f851fa3e024b" containerName="storage-initializer"
May 26 16:33:58.738649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.738077    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="7e9903aa-c2c0-4f7b-aa46-6858e9838019" containerName="main"
May 26 16:33:58.738649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.738091    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="2338525a-5955-4d6d-9189-f851fa3e024b" containerName="storage-initializer"
May 26 16:33:58.738649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.738100    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="tokenizer"
May 26 16:33:58.738649 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.738112    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="22ffecdb-9d5e-4f6e-ad7a-7174abee325c" containerName="main"
May 26 16:33:58.740933 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.740910    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.743491 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.743467    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:33:58.743622 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.743471    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:33:58.744167 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.744146    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 26 16:33:58.744266 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.744153    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:33:58.744266 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.744202    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-fbdnv\""
May 26 16:33:58.751997 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.751974    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"]
May 26 16:33:58.813377 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.813339    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.813560 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.813380    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.813560 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.813495    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm6dx\" (UniqueName: \"kubernetes.io/projected/d00e9a8f-93e2-4750-9d36-f721d834a282-kube-api-access-lm6dx\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.813878 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.813848    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.814310 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.814273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.814483 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.814341    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00e9a8f-93e2-4750-9d36-f721d834a282-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.915324 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915283    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lm6dx\" (UniqueName: \"kubernetes.io/projected/d00e9a8f-93e2-4750-9d36-f721d834a282-kube-api-access-lm6dx\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.915324 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915330    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.915584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915375    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.915584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915395    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00e9a8f-93e2-4750-9d36-f721d834a282-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.915584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915419    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.915584 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915439    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.915838 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915815    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.915973 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915948    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.916053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915971    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.916053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.915999    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.918026 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.918006    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00e9a8f-93e2-4750-9d36-f721d834a282-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:58.923355 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:58.923331    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm6dx\" (UniqueName: \"kubernetes.io/projected/d00e9a8f-93e2-4750-9d36-f721d834a282-kube-api-access-lm6dx\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-4j559\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:59.050107 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:59.050068    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:33:59.181278 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:59.181237    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"]
May 26 16:33:59.183668 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:33:59.183635    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00e9a8f_93e2_4750_9d36_f721d834a282.slice/crio-933e52e394f4004d888a5c336f4c8aafa9fba892c956db41ed17b78abb9afe43 WatchSource:0}: Error finding container 933e52e394f4004d888a5c336f4c8aafa9fba892c956db41ed17b78abb9afe43: Status 404 returned error can't find the container with id 933e52e394f4004d888a5c336f4c8aafa9fba892c956db41ed17b78abb9afe43
May 26 16:33:59.313032 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:59.312929    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" event={"ID":"d00e9a8f-93e2-4750-9d36-f721d834a282","Type":"ContainerStarted","Data":"05e21e0589d589c3197743d91dc097662f7c94ab59ed487b469ac4db8a0b6766"}
May 26 16:33:59.313032 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:33:59.312972    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" event={"ID":"d00e9a8f-93e2-4750-9d36-f721d834a282","Type":"ContainerStarted","Data":"933e52e394f4004d888a5c336f4c8aafa9fba892c956db41ed17b78abb9afe43"}
May 26 16:34:00.317541 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:00.317503    2561 generic.go:358] "Generic (PLEG): container finished" podID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerID="05e21e0589d589c3197743d91dc097662f7c94ab59ed487b469ac4db8a0b6766" exitCode=0
May 26 16:34:00.317939 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:00.317586    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" event={"ID":"d00e9a8f-93e2-4750-9d36-f721d834a282","Type":"ContainerDied","Data":"05e21e0589d589c3197743d91dc097662f7c94ab59ed487b469ac4db8a0b6766"}
May 26 16:34:01.322534 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:01.322499    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" event={"ID":"d00e9a8f-93e2-4750-9d36-f721d834a282","Type":"ContainerStarted","Data":"cd3660d25342a63e4d9e3c28883085bb26a295374a959aa9e070f4e2e9869bcb"}
May 26 16:34:01.322534 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:01.322540    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" event={"ID":"d00e9a8f-93e2-4750-9d36-f721d834a282","Type":"ContainerStarted","Data":"35e68a2551fcc1f20524c437841881d746104cb0eee35e13e659c79d1fa9bbca"}
May 26 16:34:01.323127 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:01.322584    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:34:01.345374 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:01.345310    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" podStartSLOduration=3.3452933639999998 podStartE2EDuration="3.345293364s" podCreationTimestamp="2026-05-26 16:33:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:34:01.343444279 +0000 UTC m=+1988.215702714" watchObservedRunningTime="2026-05-26 16:34:01.345293364 +0000 UTC m=+1988.217551798"
May 26 16:34:09.050996 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:09.050947    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:34:09.050996 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:09.051006    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:34:09.053679 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:09.053657    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:34:09.351790 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:09.351712    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:34:30.355997 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:34:30.355961    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:35:53.743084 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:35:53.743054    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:35:53.745667 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:35:53.745648    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:38:19.965590 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:19.965499    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"]
May 26 16:38:19.966138 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:19.965862    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="main" containerID="cri-o://35e68a2551fcc1f20524c437841881d746104cb0eee35e13e659c79d1fa9bbca" gracePeriod=30
May 26 16:38:19.966138 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:19.965960    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="tokenizer" containerID="cri-o://cd3660d25342a63e4d9e3c28883085bb26a295374a959aa9e070f4e2e9869bcb" gracePeriod=30
May 26 16:38:20.167260 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:20.167227    2561 generic.go:358] "Generic (PLEG): container finished" podID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerID="35e68a2551fcc1f20524c437841881d746104cb0eee35e13e659c79d1fa9bbca" exitCode=0
May 26 16:38:20.167426 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:20.167304    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" event={"ID":"d00e9a8f-93e2-4750-9d36-f721d834a282","Type":"ContainerDied","Data":"35e68a2551fcc1f20524c437841881d746104cb0eee35e13e659c79d1fa9bbca"}
May 26 16:38:20.354501 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:38:20.354473    2561 logging.go:55] [core] [Channel #770 SubChannel #771]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.36:9003", ServerName: "10.134.0.36:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.36:9003: connect: connection refused"
May 26 16:38:21.172913 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.172857    2561 generic.go:358] "Generic (PLEG): container finished" podID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerID="cd3660d25342a63e4d9e3c28883085bb26a295374a959aa9e070f4e2e9869bcb" exitCode=0
May 26 16:38:21.173265 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.172932    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" event={"ID":"d00e9a8f-93e2-4750-9d36-f721d834a282","Type":"ContainerDied","Data":"cd3660d25342a63e4d9e3c28883085bb26a295374a959aa9e070f4e2e9869bcb"}
May 26 16:38:21.217608 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.217582    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:38:21.291445 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.291414    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm6dx\" (UniqueName: \"kubernetes.io/projected/d00e9a8f-93e2-4750-9d36-f721d834a282-kube-api-access-lm6dx\") pod \"d00e9a8f-93e2-4750-9d36-f721d834a282\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") "
May 26 16:38:21.291603 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.291497    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-cache\") pod \"d00e9a8f-93e2-4750-9d36-f721d834a282\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") "
May 26 16:38:21.291603 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.291544    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00e9a8f-93e2-4750-9d36-f721d834a282-tls-certs\") pod \"d00e9a8f-93e2-4750-9d36-f721d834a282\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") "
May 26 16:38:21.291603 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.291596    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-kserve-provision-location\") pod \"d00e9a8f-93e2-4750-9d36-f721d834a282\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") "
May 26 16:38:21.291751 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.291620    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-uds\") pod \"d00e9a8f-93e2-4750-9d36-f721d834a282\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") "
May 26 16:38:21.291751 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.291650    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-tmp\") pod \"d00e9a8f-93e2-4750-9d36-f721d834a282\" (UID: \"d00e9a8f-93e2-4750-9d36-f721d834a282\") "
May 26 16:38:21.291860 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.291792    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "d00e9a8f-93e2-4750-9d36-f721d834a282" (UID: "d00e9a8f-93e2-4750-9d36-f721d834a282"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:21.291959 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.291924    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d00e9a8f-93e2-4750-9d36-f721d834a282" (UID: "d00e9a8f-93e2-4750-9d36-f721d834a282"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:21.292025 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.292010    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "d00e9a8f-93e2-4750-9d36-f721d834a282" (UID: "d00e9a8f-93e2-4750-9d36-f721d834a282"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:21.292092 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.292024    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:38:21.292092 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.292050    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-uds\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:38:21.292399 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.292375    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d00e9a8f-93e2-4750-9d36-f721d834a282" (UID: "d00e9a8f-93e2-4750-9d36-f721d834a282"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:38:21.293555 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.293536    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00e9a8f-93e2-4750-9d36-f721d834a282-kube-api-access-lm6dx" (OuterVolumeSpecName: "kube-api-access-lm6dx") pod "d00e9a8f-93e2-4750-9d36-f721d834a282" (UID: "d00e9a8f-93e2-4750-9d36-f721d834a282"). InnerVolumeSpecName "kube-api-access-lm6dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:38:21.293637 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.293593    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00e9a8f-93e2-4750-9d36-f721d834a282-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d00e9a8f-93e2-4750-9d36-f721d834a282" (UID: "d00e9a8f-93e2-4750-9d36-f721d834a282"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:38:21.354714 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.354681    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.36:9003\" within 1s: context deadline exceeded"
May 26 16:38:21.392919 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.392839    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-kserve-provision-location\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:38:21.392919 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.392863    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d00e9a8f-93e2-4750-9d36-f721d834a282-tokenizer-tmp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:38:21.392919 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.392873    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm6dx\" (UniqueName: \"kubernetes.io/projected/d00e9a8f-93e2-4750-9d36-f721d834a282-kube-api-access-lm6dx\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:38:21.392919 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:21.392883    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d00e9a8f-93e2-4750-9d36-f721d834a282-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:38:22.178687 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:22.178651    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559" event={"ID":"d00e9a8f-93e2-4750-9d36-f721d834a282","Type":"ContainerDied","Data":"933e52e394f4004d888a5c336f4c8aafa9fba892c956db41ed17b78abb9afe43"}
May 26 16:38:22.178687 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:22.178686    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"
May 26 16:38:22.179197 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:22.178701    2561 scope.go:117] "RemoveContainer" containerID="cd3660d25342a63e4d9e3c28883085bb26a295374a959aa9e070f4e2e9869bcb"
May 26 16:38:22.186593 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:22.186576    2561 scope.go:117] "RemoveContainer" containerID="35e68a2551fcc1f20524c437841881d746104cb0eee35e13e659c79d1fa9bbca"
May 26 16:38:22.193926 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:22.193908    2561 scope.go:117] "RemoveContainer" containerID="05e21e0589d589c3197743d91dc097662f7c94ab59ed487b469ac4db8a0b6766"
May 26 16:38:22.205085 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:22.205057    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"]
May 26 16:38:22.212789 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:22.212762    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-4j559"]
May 26 16:38:23.712295 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:23.712252    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" path="/var/lib/kubelet/pods/d00e9a8f-93e2-4750-9d36-f721d834a282/volumes"
May 26 16:38:35.838432 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.838398    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"]
May 26 16:38:35.838999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.838826    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="tokenizer"
May 26 16:38:35.838999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.838845    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="tokenizer"
May 26 16:38:35.838999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.838862    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="storage-initializer"
May 26 16:38:35.838999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.838871    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="storage-initializer"
May 26 16:38:35.838999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.838883    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="main"
May 26 16:38:35.838999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.838908    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="main"
May 26 16:38:35.838999 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.838998    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="tokenizer"
May 26 16:38:35.839375 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.839013    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="d00e9a8f-93e2-4750-9d36-f721d834a282" containerName="main"
May 26 16:38:35.842397 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.842375    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:35.845112 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.845090    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 26 16:38:35.845264 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.845237    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 26 16:38:35.845992 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.845964    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8m4rr\""
May 26 16:38:35.845992 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.845975    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
May 26 16:38:35.845992 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.845964    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-epp-sa-dockercfg-lb46f\""
May 26 16:38:35.853823 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:35.853801    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"]
May 26 16:38:36.017542 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.017504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.017710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.017565    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875ef8b0-4483-4104-8a96-c050dd1117d9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.017710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.017634    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9wk7\" (UniqueName: \"kubernetes.io/projected/875ef8b0-4483-4104-8a96-c050dd1117d9-kube-api-access-h9wk7\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.017710 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.017700    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.017819 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.017733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.017819 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.017763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119084 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.118993    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119084 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119037    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119294 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119091    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119294 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119147    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875ef8b0-4483-4104-8a96-c050dd1117d9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119294 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119174    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h9wk7\" (UniqueName: \"kubernetes.io/projected/875ef8b0-4483-4104-8a96-c050dd1117d9-kube-api-access-h9wk7\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119294 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119215    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119511 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119467    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-tmp\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119568 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119518    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-cache\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119646 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119624    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-uds\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.119696 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.119661    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-kserve-provision-location\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.121653 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.121629    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875ef8b0-4483-4104-8a96-c050dd1117d9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.128053 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.128030    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9wk7\" (UniqueName: \"kubernetes.io/projected/875ef8b0-4483-4104-8a96-c050dd1117d9-kube-api-access-h9wk7\") pod \"stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.154460 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.154433    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:36.287606 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.287580    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"]
May 26 16:38:36.289872 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:38:36.289848    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875ef8b0_4483_4104_8a96_c050dd1117d9.slice/crio-f9ae37101e5ef42ed8ac901c5465dc3a549d5c5f3f51aaa304f1d46d2f9eccaa WatchSource:0}: Error finding container f9ae37101e5ef42ed8ac901c5465dc3a549d5c5f3f51aaa304f1d46d2f9eccaa: Status 404 returned error can't find the container with id f9ae37101e5ef42ed8ac901c5465dc3a549d5c5f3f51aaa304f1d46d2f9eccaa
May 26 16:38:36.292341 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:36.292322    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:38:37.229826 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:37.229789    2561 generic.go:358] "Generic (PLEG): container finished" podID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerID="7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537" exitCode=0
May 26 16:38:37.230211 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:37.229853    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" event={"ID":"875ef8b0-4483-4104-8a96-c050dd1117d9","Type":"ContainerDied","Data":"7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537"}
May 26 16:38:37.230211 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:37.229881    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" event={"ID":"875ef8b0-4483-4104-8a96-c050dd1117d9","Type":"ContainerStarted","Data":"f9ae37101e5ef42ed8ac901c5465dc3a549d5c5f3f51aaa304f1d46d2f9eccaa"}
May 26 16:38:38.236204 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:38.236167    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" event={"ID":"875ef8b0-4483-4104-8a96-c050dd1117d9","Type":"ContainerStarted","Data":"028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333"}
May 26 16:38:38.236204 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:38.236203    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" event={"ID":"875ef8b0-4483-4104-8a96-c050dd1117d9","Type":"ContainerStarted","Data":"09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341"}
May 26 16:38:38.236755 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:38.236269    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:38.258672 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:38.258603    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" podStartSLOduration=3.258590141 podStartE2EDuration="3.258590141s" podCreationTimestamp="2026-05-26 16:38:35 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:38:38.258054258 +0000 UTC m=+2265.130312693" watchObservedRunningTime="2026-05-26 16:38:38.258590141 +0000 UTC m=+2265.130848576"
May 26 16:38:46.155193 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:46.155160    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:46.155791 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:46.155331    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:46.158080 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:46.158054    2561 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:38:46.265563 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:38:46.265535    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:39:08.274482 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:39:08.274450    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:40:53.766717 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:40:53.766685    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:40:53.770163 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:40:53.770141    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:41:57.030231 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:57.030192    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"]
May 26 16:41:57.031022 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:57.030979    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="tokenizer" containerID="cri-o://028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333" gracePeriod=30
May 26 16:41:57.031355 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:57.031288    2561 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="main" containerID="cri-o://09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341" gracePeriod=30
May 26 16:41:57.892756 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:57.892721    2561 generic.go:358] "Generic (PLEG): container finished" podID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerID="09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341" exitCode=0
May 26 16:41:57.892952 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:57.892807    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" event={"ID":"875ef8b0-4483-4104-8a96-c050dd1117d9","Type":"ContainerDied","Data":"09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341"}
May 26 16:41:58.276977 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.276954    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:41:58.331565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.331537    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-tmp\") pod \"875ef8b0-4483-4104-8a96-c050dd1117d9\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") "
May 26 16:41:58.331743 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.331572    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-cache\") pod \"875ef8b0-4483-4104-8a96-c050dd1117d9\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") "
May 26 16:41:58.331743 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.331609    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9wk7\" (UniqueName: \"kubernetes.io/projected/875ef8b0-4483-4104-8a96-c050dd1117d9-kube-api-access-h9wk7\") pod \"875ef8b0-4483-4104-8a96-c050dd1117d9\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") "
May 26 16:41:58.331743 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.331655    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-kserve-provision-location\") pod \"875ef8b0-4483-4104-8a96-c050dd1117d9\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") "
May 26 16:41:58.331743 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.331714    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-uds\") pod \"875ef8b0-4483-4104-8a96-c050dd1117d9\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") "
May 26 16:41:58.331982 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.331746    2561 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875ef8b0-4483-4104-8a96-c050dd1117d9-tls-certs\") pod \"875ef8b0-4483-4104-8a96-c050dd1117d9\" (UID: \"875ef8b0-4483-4104-8a96-c050dd1117d9\") "
May 26 16:41:58.331982 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.331922    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "875ef8b0-4483-4104-8a96-c050dd1117d9" (UID: "875ef8b0-4483-4104-8a96-c050dd1117d9"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:58.332071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.331980    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-tmp" (OuterVolumeSpecName: "tokenizer-tmp") pod "875ef8b0-4483-4104-8a96-c050dd1117d9" (UID: "875ef8b0-4483-4104-8a96-c050dd1117d9"). InnerVolumeSpecName "tokenizer-tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:58.332071 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.332026    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "875ef8b0-4483-4104-8a96-c050dd1117d9" (UID: "875ef8b0-4483-4104-8a96-c050dd1117d9"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:58.332168 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.332095    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-tmp\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-tmp\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:41:58.332168 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.332111    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-cache\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:41:58.332512 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.332485    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "875ef8b0-4483-4104-8a96-c050dd1117d9" (UID: "875ef8b0-4483-4104-8a96-c050dd1117d9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 26 16:41:58.333874 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.333853    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875ef8b0-4483-4104-8a96-c050dd1117d9-kube-api-access-h9wk7" (OuterVolumeSpecName: "kube-api-access-h9wk7") pod "875ef8b0-4483-4104-8a96-c050dd1117d9" (UID: "875ef8b0-4483-4104-8a96-c050dd1117d9"). InnerVolumeSpecName "kube-api-access-h9wk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 26 16:41:58.333996 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.333972    2561 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875ef8b0-4483-4104-8a96-c050dd1117d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "875ef8b0-4483-4104-8a96-c050dd1117d9" (UID: "875ef8b0-4483-4104-8a96-c050dd1117d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 26 16:41:58.432831 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.432737    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-kserve-provision-location\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:41:58.432831 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.432770    2561 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/875ef8b0-4483-4104-8a96-c050dd1117d9-tokenizer-uds\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:41:58.432831 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.432784    2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875ef8b0-4483-4104-8a96-c050dd1117d9-tls-certs\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:41:58.432831 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.432796    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9wk7\" (UniqueName: \"kubernetes.io/projected/875ef8b0-4483-4104-8a96-c050dd1117d9-kube-api-access-h9wk7\") on node \"ip-10-0-133-131.ec2.internal\" DevicePath \"\""
May 26 16:41:58.898506 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.898475    2561 generic.go:358] "Generic (PLEG): container finished" podID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerID="028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333" exitCode=0
May 26 16:41:58.898698 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.898537    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" event={"ID":"875ef8b0-4483-4104-8a96-c050dd1117d9","Type":"ContainerDied","Data":"028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333"}
May 26 16:41:58.898698 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.898546    2561 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"
May 26 16:41:58.898698 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.898566    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" event={"ID":"875ef8b0-4483-4104-8a96-c050dd1117d9","Type":"ContainerDied","Data":"f9ae37101e5ef42ed8ac901c5465dc3a549d5c5f3f51aaa304f1d46d2f9eccaa"}
May 26 16:41:58.898698 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.898586    2561 scope.go:117] "RemoveContainer" containerID="028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333"
May 26 16:41:58.906906 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.906869    2561 scope.go:117] "RemoveContainer" containerID="09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341"
May 26 16:41:58.917322 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.917299    2561 scope.go:117] "RemoveContainer" containerID="7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537"
May 26 16:41:58.922669 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.922642    2561 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"]
May 26 16:41:58.925091 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.925071    2561 scope.go:117] "RemoveContainer" containerID="028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333"
May 26 16:41:58.925636 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:41:58.925607    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333\": container with ID starting with 028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333 not found: ID does not exist" containerID="028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333"
May 26 16:41:58.925727 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.925647    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333"} err="failed to get container status \"028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333\": rpc error: code = NotFound desc = could not find container \"028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333\": container with ID starting with 028aaaf0e8f3e55d6d3a9de10f20023803b8ae6143bd2aee89113266bd005333 not found: ID does not exist"
May 26 16:41:58.925727 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.925676    2561 scope.go:117] "RemoveContainer" containerID="09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341"
May 26 16:41:58.926224 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:41:58.926204    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341\": container with ID starting with 09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341 not found: ID does not exist" containerID="09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341"
May 26 16:41:58.926303 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.926230    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341"} err="failed to get container status \"09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341\": rpc error: code = NotFound desc = could not find container \"09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341\": container with ID starting with 09185e61f61005dd3a08961905b9e1526795c13800d9f43d7d5f05cde4195341 not found: ID does not exist"
May 26 16:41:58.926303 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.926247    2561 scope.go:117] "RemoveContainer" containerID="7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537"
May 26 16:41:58.926514 ip-10-0-133-131 kubenswrapper[2561]: E0526 16:41:58.926489    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537\": container with ID starting with 7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537 not found: ID does not exist" containerID="7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537"
May 26 16:41:58.926565 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.926522    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537"} err="failed to get container status \"7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537\": rpc error: code = NotFound desc = could not find container \"7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537\": container with ID starting with 7d68a43836eb8302a6617c508deef501450ed360ff7e8d127782c209c3480537 not found: ID does not exist"
May 26 16:41:58.927362 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:58.927342    2561 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw"]
May 26 16:41:59.273835 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.273791    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-router-scheduler-5549f64b68-t6bnw" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="main" probeResult="failure" output="timeout: failed to connect service \"10.134.0.37:9003\" within 1s: context deadline exceeded"
May 26 16:41:59.274033 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:41:59.273875    2561 logging.go:55] [core] [Channel #883 SubChannel #884]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.37:9003", ServerName: "10.134.0.37:9003", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.37:9003: operation was canceled"
May 26 16:41:59.279945 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.279922    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-565f67c69d-p8h6l"]
May 26 16:41:59.280293 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.280222    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="tokenizer"
May 26 16:41:59.280293 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.280232    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="tokenizer"
May 26 16:41:59.280293 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.280241    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="storage-initializer"
May 26 16:41:59.280293 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.280246    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="storage-initializer"
May 26 16:41:59.280293 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.280262    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="main"
May 26 16:41:59.280293 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.280268    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="main"
May 26 16:41:59.280526 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.280318    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="main"
May 26 16:41:59.280526 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.280328    2561 memory_manager.go:356] "RemoveStaleState removing state" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" containerName="tokenizer"
May 26 16:41:59.284949 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.284931    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:41:59.290380 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.290359    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-djnd8\""
May 26 16:41:59.290504 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.290482    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
May 26 16:41:59.296232 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.296210    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-565f67c69d-p8h6l"]
May 26 16:41:59.340592 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.340560    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfwj\" (UniqueName: \"kubernetes.io/projected/4a429e3f-44c8-45df-97f9-1938ef2a1a8f-kube-api-access-wpfwj\") pod \"llmisvc-controller-manager-565f67c69d-p8h6l\" (UID: \"4a429e3f-44c8-45df-97f9-1938ef2a1a8f\") " pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:41:59.340726 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.340615    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a429e3f-44c8-45df-97f9-1938ef2a1a8f-cert\") pod \"llmisvc-controller-manager-565f67c69d-p8h6l\" (UID: \"4a429e3f-44c8-45df-97f9-1938ef2a1a8f\") " pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:41:59.442022 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.441988    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a429e3f-44c8-45df-97f9-1938ef2a1a8f-cert\") pod \"llmisvc-controller-manager-565f67c69d-p8h6l\" (UID: \"4a429e3f-44c8-45df-97f9-1938ef2a1a8f\") " pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:41:59.442177 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.442059    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfwj\" (UniqueName: \"kubernetes.io/projected/4a429e3f-44c8-45df-97f9-1938ef2a1a8f-kube-api-access-wpfwj\") pod \"llmisvc-controller-manager-565f67c69d-p8h6l\" (UID: \"4a429e3f-44c8-45df-97f9-1938ef2a1a8f\") " pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:41:59.444337 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.444313    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4a429e3f-44c8-45df-97f9-1938ef2a1a8f-cert\") pod \"llmisvc-controller-manager-565f67c69d-p8h6l\" (UID: \"4a429e3f-44c8-45df-97f9-1938ef2a1a8f\") " pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:41:59.450664 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.450636    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfwj\" (UniqueName: \"kubernetes.io/projected/4a429e3f-44c8-45df-97f9-1938ef2a1a8f-kube-api-access-wpfwj\") pod \"llmisvc-controller-manager-565f67c69d-p8h6l\" (UID: \"4a429e3f-44c8-45df-97f9-1938ef2a1a8f\") " pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:41:59.594479 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.594390    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:41:59.713315 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.713272    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="875ef8b0-4483-4104-8a96-c050dd1117d9" path="/var/lib/kubelet/pods/875ef8b0-4483-4104-8a96-c050dd1117d9/volumes"
May 26 16:41:59.725900 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.725868    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-565f67c69d-p8h6l"]
May 26 16:41:59.728348 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:41:59.728319    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4a429e3f_44c8_45df_97f9_1938ef2a1a8f.slice/crio-cacd1d20d940a86f454d66bf1079abc2d02f8fc64e216fb340aca58557d18efb WatchSource:0}: Error finding container cacd1d20d940a86f454d66bf1079abc2d02f8fc64e216fb340aca58557d18efb: Status 404 returned error can't find the container with id cacd1d20d940a86f454d66bf1079abc2d02f8fc64e216fb340aca58557d18efb
May 26 16:41:59.902427 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:41:59.902332    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l" event={"ID":"4a429e3f-44c8-45df-97f9-1938ef2a1a8f","Type":"ContainerStarted","Data":"cacd1d20d940a86f454d66bf1079abc2d02f8fc64e216fb340aca58557d18efb"}
May 26 16:42:03.918906 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:42:03.918850    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l" event={"ID":"4a429e3f-44c8-45df-97f9-1938ef2a1a8f","Type":"ContainerStarted","Data":"e97410bf3148cfc4b4ae49d6a92d0010c17e605dca89a1a5fdc9b5b63af873ff"}
May 26 16:42:03.919327 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:42:03.919046    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:42:03.936965 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:42:03.936909    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l" podStartSLOduration=1.850276455 podStartE2EDuration="4.936867961s" podCreationTimestamp="2026-05-26 16:41:59 +0000 UTC" firstStartedPulling="2026-05-26 16:41:59.729620076 +0000 UTC m=+2466.601878492" lastFinishedPulling="2026-05-26 16:42:02.816211573 +0000 UTC m=+2469.688469998" observedRunningTime="2026-05-26 16:42:03.936297355 +0000 UTC m=+2470.808555790" watchObservedRunningTime="2026-05-26 16:42:03.936867961 +0000 UTC m=+2470.809126396"
May 26 16:42:34.925607 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:42:34.925576    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-565f67c69d-p8h6l"
May 26 16:43:36.399136 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:36.399106    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8488b484-56l4v_6c9859ce-6c0b-4dad-a6e9-4b672695357a/router/0.log"
May 26 16:43:37.303738 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:37.303710    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8488b484-56l4v_6c9859ce-6c0b-4dad-a6e9-4b672695357a/router/0.log"
May 26 16:43:38.111161 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:38.111109    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-488m8_ee83b144-e776-4023-a729-91d0f4292b86/authorino/0.log"
May 26 16:43:38.201339 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:38.201309    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-pjll2_ca33e181-a488-43f6-ac69-b8043f9024f4/manager/0.log"
May 26 16:43:40.639751 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.639717    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9f8jk/must-gather-cqtkn"]
May 26 16:43:40.642212 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.642194    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f8jk/must-gather-cqtkn"
May 26 16:43:40.645284 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.645265    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f8jk\"/\"kube-root-ca.crt\""
May 26 16:43:40.645789 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.645771    2561 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9f8jk\"/\"openshift-service-ca.crt\""
May 26 16:43:40.645960 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.645944    2561 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9f8jk\"/\"default-dockercfg-bg6lc\""
May 26 16:43:40.653154 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.653127    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f8jk/must-gather-cqtkn"]
May 26 16:43:40.705776 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.705749    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/254fe041-d591-4890-b4d4-dd4347eeda46-must-gather-output\") pod \"must-gather-cqtkn\" (UID: \"254fe041-d591-4890-b4d4-dd4347eeda46\") " pod="openshift-must-gather-9f8jk/must-gather-cqtkn"
May 26 16:43:40.705941 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.705783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dtwr\" (UniqueName: \"kubernetes.io/projected/254fe041-d591-4890-b4d4-dd4347eeda46-kube-api-access-5dtwr\") pod \"must-gather-cqtkn\" (UID: \"254fe041-d591-4890-b4d4-dd4347eeda46\") " pod="openshift-must-gather-9f8jk/must-gather-cqtkn"
May 26 16:43:40.807086 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.807051    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5dtwr\" (UniqueName: \"kubernetes.io/projected/254fe041-d591-4890-b4d4-dd4347eeda46-kube-api-access-5dtwr\") pod \"must-gather-cqtkn\" (UID: \"254fe041-d591-4890-b4d4-dd4347eeda46\") " pod="openshift-must-gather-9f8jk/must-gather-cqtkn"
May 26 16:43:40.807262 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.807176    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/254fe041-d591-4890-b4d4-dd4347eeda46-must-gather-output\") pod \"must-gather-cqtkn\" (UID: \"254fe041-d591-4890-b4d4-dd4347eeda46\") " pod="openshift-must-gather-9f8jk/must-gather-cqtkn"
May 26 16:43:40.807510 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.807489    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/254fe041-d591-4890-b4d4-dd4347eeda46-must-gather-output\") pod \"must-gather-cqtkn\" (UID: \"254fe041-d591-4890-b4d4-dd4347eeda46\") " pod="openshift-must-gather-9f8jk/must-gather-cqtkn"
May 26 16:43:40.817796 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.817772    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dtwr\" (UniqueName: \"kubernetes.io/projected/254fe041-d591-4890-b4d4-dd4347eeda46-kube-api-access-5dtwr\") pod \"must-gather-cqtkn\" (UID: \"254fe041-d591-4890-b4d4-dd4347eeda46\") " pod="openshift-must-gather-9f8jk/must-gather-cqtkn"
May 26 16:43:40.952343 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:40.952269    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f8jk/must-gather-cqtkn"
May 26 16:43:41.078438 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:41.078400    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f8jk/must-gather-cqtkn"]
May 26 16:43:41.082067 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:43:41.082043    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod254fe041_d591_4890_b4d4_dd4347eeda46.slice/crio-24dbd8a272790467d73fa743ec08b2d7799bbc16dc0ee6a85132b9813513636a WatchSource:0}: Error finding container 24dbd8a272790467d73fa743ec08b2d7799bbc16dc0ee6a85132b9813513636a: Status 404 returned error can't find the container with id 24dbd8a272790467d73fa743ec08b2d7799bbc16dc0ee6a85132b9813513636a
May 26 16:43:41.083839 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:41.083816    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 26 16:43:41.246012 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:41.245917    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/must-gather-cqtkn" event={"ID":"254fe041-d591-4890-b4d4-dd4347eeda46","Type":"ContainerStarted","Data":"24dbd8a272790467d73fa743ec08b2d7799bbc16dc0ee6a85132b9813513636a"}
May 26 16:43:42.259671 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:42.259597    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/must-gather-cqtkn" event={"ID":"254fe041-d591-4890-b4d4-dd4347eeda46","Type":"ContainerStarted","Data":"4e647cb21e43c7d62d5dececb3c3cc962fc0c057636dacae3c2307ab3cd54362"}
May 26 16:43:42.259671 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:42.259630    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/must-gather-cqtkn" event={"ID":"254fe041-d591-4890-b4d4-dd4347eeda46","Type":"ContainerStarted","Data":"5395c887ef29128064d25901577c5e5361b320a267159cf4198d71123803eaae"}
May 26 16:43:42.277878 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:42.277807    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9f8jk/must-gather-cqtkn" podStartSLOduration=1.487678468 podStartE2EDuration="2.277787894s" podCreationTimestamp="2026-05-26 16:43:40 +0000 UTC" firstStartedPulling="2026-05-26 16:43:41.084034739 +0000 UTC m=+2567.956293154" lastFinishedPulling="2026-05-26 16:43:41.87414415 +0000 UTC m=+2568.746402580" observedRunningTime="2026-05-26 16:43:42.276989476 +0000 UTC m=+2569.149247910" watchObservedRunningTime="2026-05-26 16:43:42.277787894 +0000 UTC m=+2569.150046329"
May 26 16:43:43.458092 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:43.458058    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-b78wx_b5eb3a03-f201-4bb2-9777-e4cdf7db49d0/global-pull-secret-syncer/0.log"
May 26 16:43:43.591452 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:43.591418    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c8tfr_61d768a5-d28e-4ac8-8d64-9a8ab4718e68/konnectivity-agent/0.log"
May 26 16:43:43.685145 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:43.685113    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-131.ec2.internal_e74c204e9201dd85fbf7a07770c47c82/haproxy/0.log"
May 26 16:43:47.980005 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:47.979917    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68bd676465-488m8_ee83b144-e776-4023-a729-91d0f4292b86/authorino/0.log"
May 26 16:43:48.147787 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:48.147758    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57dd85bb4-pjll2_ca33e181-a488-43f6-ac69-b8043f9024f4/manager/0.log"
May 26 16:43:49.411605 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:49.411568    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-7dmc6_2bae3ffc-8cbc-46c6-b79f-f0bd40f70940/cluster-monitoring-operator/0.log"
May 26 16:43:49.454756 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:49.454718    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-sq547_74a23d36-6c2e-405e-83d9-9b6ca1ac9185/kube-state-metrics/0.log"
May 26 16:43:49.478258 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:49.478217    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-sq547_74a23d36-6c2e-405e-83d9-9b6ca1ac9185/kube-rbac-proxy-main/0.log"
May 26 16:43:49.503572 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:49.503511    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-sq547_74a23d36-6c2e-405e-83d9-9b6ca1ac9185/kube-rbac-proxy-self/0.log"
May 26 16:43:49.539589 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:49.539547    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-59874dc46b-hvndz_20728c3e-d771-4696-9ab9-04a53ada173f/metrics-server/0.log"
May 26 16:43:49.856067 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:49.856021    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqf57_084ed376-fc35-4e25-ac53-404b0d311e87/node-exporter/0.log"
May 26 16:43:49.888520 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:49.888491    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqf57_084ed376-fc35-4e25-ac53-404b0d311e87/kube-rbac-proxy/0.log"
May 26 16:43:49.915408 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:49.915366    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qqf57_084ed376-fc35-4e25-ac53-404b0d311e87/init-textfile/0.log"
May 26 16:43:50.053836 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:50.053809    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b9b463f-5708-4999-990f-ad5d7be3c697/prometheus/0.log"
May 26 16:43:50.079409 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:50.079331    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b9b463f-5708-4999-990f-ad5d7be3c697/config-reloader/0.log"
May 26 16:43:50.108013 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:50.107983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b9b463f-5708-4999-990f-ad5d7be3c697/thanos-sidecar/0.log"
May 26 16:43:50.133812 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:50.133772    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b9b463f-5708-4999-990f-ad5d7be3c697/kube-rbac-proxy-web/0.log"
May 26 16:43:50.161225 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:50.161174    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b9b463f-5708-4999-990f-ad5d7be3c697/kube-rbac-proxy/0.log"
May 26 16:43:50.186914 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:50.186866    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b9b463f-5708-4999-990f-ad5d7be3c697/kube-rbac-proxy-thanos/0.log"
May 26 16:43:50.214906 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:50.214861    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3b9b463f-5708-4999-990f-ad5d7be3c697/init-config-reloader/0.log"
May 26 16:43:51.739880 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:51.739850    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-78ktw_bf1ccb53-7cd2-4015-bc33-36ac74bc22b5/networking-console-plugin/0.log"
May 26 16:43:52.539009 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.538973    2561 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"]
May 26 16:43:52.543904 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.543863    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.551260 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.551230    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"]
May 26 16:43:52.623857 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.623822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-proc\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.624202 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.624181    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-lib-modules\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.624450 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.624431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-podres\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.624606 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.624591    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-sys\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.624752 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.624735    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpjcj\" (UniqueName: \"kubernetes.io/projected/7420c1f6-864a-4402-84a4-402a1b03cb6e-kube-api-access-kpjcj\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725716 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725687    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-podres\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725716 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725720    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-sys\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725738    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kpjcj\" (UniqueName: \"kubernetes.io/projected/7420c1f6-864a-4402-84a4-402a1b03cb6e-kube-api-access-kpjcj\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725780    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-proc\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725819    2561 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-lib-modules\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725842    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-sys\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725862    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-podres\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725952    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-proc\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.725993 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.725969    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7420c1f6-864a-4402-84a4-402a1b03cb6e-lib-modules\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.734927 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.734902    2561 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpjcj\" (UniqueName: \"kubernetes.io/projected/7420c1f6-864a-4402-84a4-402a1b03cb6e-kube-api-access-kpjcj\") pod \"perf-node-gather-daemonset-vnzfs\" (UID: \"7420c1f6-864a-4402-84a4-402a1b03cb6e\") " pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:52.859465 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:52.858913    2561 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:53.013207 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:53.013174    2561 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"]
May 26 16:43:53.016018 ip-10-0-133-131 kubenswrapper[2561]: W0526 16:43:53.015994    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7420c1f6_864a_4402_84a4_402a1b03cb6e.slice/crio-cbdfbf417d8db211fc37206edeae05d4108340e2f4acd0f8d90abf52ffea3f93 WatchSource:0}: Error finding container cbdfbf417d8db211fc37206edeae05d4108340e2f4acd0f8d90abf52ffea3f93: Status 404 returned error can't find the container with id cbdfbf417d8db211fc37206edeae05d4108340e2f4acd0f8d90abf52ffea3f93
May 26 16:43:53.314965 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:53.314926    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs" event={"ID":"7420c1f6-864a-4402-84a4-402a1b03cb6e","Type":"ContainerStarted","Data":"a603e91c48756c272498d236d63884a2334591bb9bc69211223c475ae8fa3ec9"}
May 26 16:43:53.314965 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:53.314971    2561 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs" event={"ID":"7420c1f6-864a-4402-84a4-402a1b03cb6e","Type":"ContainerStarted","Data":"cbdfbf417d8db211fc37206edeae05d4108340e2f4acd0f8d90abf52ffea3f93"}
May 26 16:43:53.315181 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:53.315088    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:53.332161 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:53.332098    2561 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs" podStartSLOduration=1.332080658 podStartE2EDuration="1.332080658s" podCreationTimestamp="2026-05-26 16:43:52 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-26 16:43:53.330473594 +0000 UTC m=+2580.202732032" watchObservedRunningTime="2026-05-26 16:43:53.332080658 +0000 UTC m=+2580.204339093"
May 26 16:43:54.180704 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:54.180676    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4xt6_78c7e49d-53fc-4af3-874c-8a7b6ab588e7/dns/0.log"
May 26 16:43:54.207470 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:54.207443    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p4xt6_78c7e49d-53fc-4af3-874c-8a7b6ab588e7/kube-rbac-proxy/0.log"
May 26 16:43:54.281790 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:54.281757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-52kjn_37120c41-44fb-484a-b6e6-f6fe6e962658/dns-node-resolver/0.log"
May 26 16:43:54.811092 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:54.811055    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c76476c85-rff5w_3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c/registry/0.log"
May 26 16:43:54.825070 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:54.825030    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5c76476c85-rff5w_3f71ece4-23ab-41d8-adcf-89f3bdc1ec5c/registry/1.log"
May 26 16:43:54.904157 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:54.904116    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wj5zg_3f685ebf-142e-4075-888a-d44a456c5812/node-ca/0.log"
May 26 16:43:55.855582 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:55.855557    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d8488b484-56l4v_6c9859ce-6c0b-4dad-a6e9-4b672695357a/router/0.log"
May 26 16:43:56.383209 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:56.383184    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4qxkt_3b9eef08-2c0d-419e-b42e-5ad0cff81e71/serve-healthcheck-canary/0.log"
May 26 16:43:56.980588 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:56.980553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7829b_03a4348a-f490-4fba-94b1-3e0e1d82c14b/kube-rbac-proxy/0.log"
May 26 16:43:57.005574 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:57.005553    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7829b_03a4348a-f490-4fba-94b1-3e0e1d82c14b/exporter/0.log"
May 26 16:43:57.037883 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:57.037861    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7829b_03a4348a-f490-4fba-94b1-3e0e1d82c14b/extractor/0.log"
May 26 16:43:59.328574 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:59.328545    2561 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9f8jk/perf-node-gather-daemonset-vnzfs"
May 26 16:43:59.708414 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:43:59.708320    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-669cc6566-4vh99_f032dba2-6b99-4f76-8e90-45f2a1c45db3/manager/0.log"
May 26 16:44:00.341076 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:00.341049    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-565f67c69d-p8h6l_4a429e3f-44c8-45df-97f9-1938ef2a1a8f/manager/0.log"
May 26 16:44:00.661291 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:00.661206    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-s2297_5218fd50-ee10-456d-aa49-f79111f8422b/manager/0.log"
May 26 16:44:07.162197 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.162165    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vn5k7_e29c2933-d508-4230-9a48-47e2758672d5/kube-multus-additional-cni-plugins/0.log"
May 26 16:44:07.191414 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.191385    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vn5k7_e29c2933-d508-4230-9a48-47e2758672d5/egress-router-binary-copy/0.log"
May 26 16:44:07.218662 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.218637    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vn5k7_e29c2933-d508-4230-9a48-47e2758672d5/cni-plugins/0.log"
May 26 16:44:07.246799 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.246773    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vn5k7_e29c2933-d508-4230-9a48-47e2758672d5/bond-cni-plugin/0.log"
May 26 16:44:07.271820 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.271793    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vn5k7_e29c2933-d508-4230-9a48-47e2758672d5/routeoverride-cni/0.log"
May 26 16:44:07.294805 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.294777    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vn5k7_e29c2933-d508-4230-9a48-47e2758672d5/whereabouts-cni-bincopy/0.log"
May 26 16:44:07.318332 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.318306    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vn5k7_e29c2933-d508-4230-9a48-47e2758672d5/whereabouts-cni/0.log"
May 26 16:44:07.356309 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.356279    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fj8z4_33633aed-0f35-47bb-9be8-158d6d96a114/kube-multus/0.log"
May 26 16:44:07.539560 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.539523    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6glcl_4fbe1d25-e242-4838-b10c-78c4a9c1196e/network-metrics-daemon/0.log"
May 26 16:44:07.563562 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:07.563535    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6glcl_4fbe1d25-e242-4838-b10c-78c4a9c1196e/kube-rbac-proxy/0.log"
May 26 16:44:08.793586 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:08.793556    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-controller/0.log"
May 26 16:44:08.815128 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:08.815104    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/0.log"
May 26 16:44:08.827049 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:08.827028    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovn-acl-logging/1.log"
May 26 16:44:08.848293 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:08.848267    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/kube-rbac-proxy-node/0.log"
May 26 16:44:08.872374 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:08.872350    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/kube-rbac-proxy-ovn-metrics/0.log"
May 26 16:44:08.893991 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:08.893969    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/northd/0.log"
May 26 16:44:08.914834 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:08.914781    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/nbdb/0.log"
May 26 16:44:08.941526 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:08.941505    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/sbdb/0.log"
May 26 16:44:09.081770 ip-10-0-133-131 kubenswrapper[2561]: I0526 16:44:09.081693    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jslmk_f9ddde5e-182c-4cd4-b95e-f80ef8fdac39/ovnkube-controller/0.log"